AUTHENTICATED , 
US. GOVERNMENT 
INFORMATION ^ 


S. Hrg. 109-76 

FEDERAL BUREAU OF INVESTIGATION’S INFORMA- 
TION TECHNOLOGY MODERNIZATION PRO- 
GRAM, TRILOGY 


HEARING 

BEFORE A 

SUBCOMMITTEE OF THE 

COMMITTEE ON APPROPRIATIONS 
UNITED STATES SENATE 

ONE HUNDRED NINTH CONGRESS 

FIRST SESSION 

SPECIAL HEARING 

FEBRUARY 3, 2005— WASHINGTON, DC 


Printed for the use of the Committee on Appropriations 



Available via the World Wide Web: http://www.access.gpo.gov/congress/senate 


U.S. GOVERNMENT PRINTING OFFICE 
20-668 PDF WASHINGTON : 2005 


For sale by the Superintendent of Documents, U.S. Government Printing Office 
Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800; DC area (202) 512-1800 
Fax: (202) 512-2250 Mail: Stop SSOP, Washington, DC 20402-0001 


COMMITTEE ON APPROPRIATIONS 


THAD COCHRAN, Mississippi, Chairman 

ROBERT C. BYRD, West Virginia 
DANIEL K. INOUYE, Hawaii 
PATRICK J. LEAHY, Vermont 
TOM HARKIN, Iowa 


TED STEVENS, Alaska 
ARLEN SPECTER, Pennsylvania 
PETE V. DOMENICI, New Mexico 
CHRISTOPHER S. BOND, Missouri 
MITCH McConnell, Kentucky 
CONRAD BURNS, Montana 
RICHARD C. SHELBY, Alabama 
JUDD GREGG, New Hampshire 
ROBERT F. BENNETT, Utah 
LARRY CRAIG, Idaho 
KAY BAILEY HUTCHISON, Texas 
MIKE DeWINE, Ohio 


BARBARA A. MIKULSKI, Maryland 
HARRY REID, Nevada 
HERB KOHL, Wisconsin 
PATTY MURRAY, Washington 
BYRON L. DORGAN, North Dakota 
DIANNE FEINSTEIN, California 
RICHARD J. DURBIN, Illinois 
TIM JOHNSON, South Dakota 
MARY L. LANDRIEU, Louisiana 


SAM BROWNBACK, Kansas 
WAYNE ALLARD, Colorado 

J. Keith Kennedy, Staff Director 
Terrence E. Sauvain, Minority Staff Director 


Subcommittee on Commerce, Justice, and State, the Judiciary, and Related 

Agencies 


JUDD GREGG, New Hampshire, Chairman 


TED STEVENS, Alaska 
PETE V. DOMENICI, New Mexico 
MITCH McConnell, Kentucky 
KAY BAILEY HUTCHISON, Texas 
SAM BROWNBACK, Kansas 
THAD COCHRAN, Mississippi 
(ex officio) 


DANIEL K. INOUYE, Hawaii 
BARBARA A. MIKULSKI, Maryland 
PATRICK J. LEAHY, Vermont 
HERB KOHL, Wisconsin 
PATTY MURRAY, Washington 
ROBERT C. BYRD, West Virginia 
(ex officio) 


Professional Staff 
Katherine Hennessey 
Dennis Balkham 
Jill Shapiro Long 
Jessica Roberts 
Nancy Perkins 
Chad Schulken (Minority) 
Kate Eltrich (Minority) 


(H) 



CONTENTS 


Page 

Opening Remarks of Senator Judd Gregg 1 

Trilogy Program Software 2 

Opening Remarks of Senator Patrick J. Leahy 2 

Possibility of Scraping Key Trilogy Components 2 

Assessments of Visual Case Files 3 

Lessons Learned 3 

Prepared Statement of Senator Patrick J. Leahy 4 

Statement of Senator Barbara A. Mikulski 6 

Technology Programs Going Bust 6 

Statement of Hon. Robert S. Mueller, III, Director, Federal Bureau of Inves- 
tigation, Department of Justice 7 

Zalmai Azmi, Chief Information Officer, Federal Bureau of Investigation, 

Department of Justice 7 

Opening Statement of Director Mueller 7 

Completed Phases of Trilogy 7 

Critical IT Improvements 8 

Technology for Street Agents 8 

Virtual Case File Answers 9 

Two-track Visual Case File Plan Adoption 10 

Responsibility for What Went Wrong 10 

Virtual Case File Funding 11 

Where Do We Go From Here 11 

Aerospace Corporation Selection 12 

Prepared Statement of Robert S. Mueller, III 13 

Quality of Personnel 19 

Cost-plus Contract and COTS Products 20 

Enterprise Architecture 21 

How Do We Get the Money Back 21 

Delivery Elements of VCF on Track 21 

Director Mueller’s Responses 22 

Federal Systems Integration and Management Request 23 

Budget to Complete Trilogy 23 

Reprogramming 24 

Recouping Funds from SAIC 24 

Case Management 25 

Information Technology Development and Funds Recovery 25 

Possibility of Scraping SAIC 26 

Decisionmaking 27 

Evaluating the 2004 Product 27 

Accelerated Funding and Oversight 29 

Independent Evaluating Assistive Team 30 

File Management and Wireless Technology 30 

Prepared Statement of Arnold L. Punaro, Executive Vice President, Science 

Applications International Corporation 33 

Prepared Statement of Gary P. Pulliam, Vice President, Civil and Commercial 

Operations, The Aerospace Corporation 40 

Prepared Statement of Glenn A. Fine, Inspector General, Department of 

Justice 60 

Prior OIG Reviews of FBI Information Technology 61 

Background on Trilogy 62 

Results of OIG Audit of Trilogy Project 62 

Causes of Trilogy’s Problems 65 

OIG Conclusions Regarding Trilogy Project 68 

Additional OIG Reviews in the FBI 68 


(III) 



IV 


Page 

Prepared Statement of Senator Charles Grassley 70 

Additional Committee Questions 71 

Questions Submitted by Senator Patrick J. Leahy 72 

Virtual Case File 72 

Terrorist Screening Center 80 



FEDERAL BUREAU OF INVESTIGATION’S IN- 
FORMATION TECHNOLOGY MODERNIZATION 
PROGRAM, TRILOGY 


THURSDAY, FEBRUARY 3, 2005 

U.S. Senate, 

Subcommittee on Commerce, Justice, and State, 

THE Judiciary, and Related Agencies, 

Committee on Appropriations, 

Washington, DC. 

The subcommittee met at 2:01 p.m., in room SD-192, Dirksen 
Senate Office Building, Hon. Judd Gregg (chairman) presiding. 

Present: Senators Gregg, Stevens, Mikulski, and Leahy. 

OPENING REMARKS OF SENATOR JUDD GREGG 

Senator Gregg. The subcommittee will come to order. I appre- 
ciate Senator Leahy being here. We haven’t really organized as an 
Appropriations Committee yet, so we do not know who is chairman 
of what and who is ranking where, but for the moment. Senator 
Leahy is serving as the acting ranking member for this sub- 
committee. It is nice to have my neighbor and friend from across 
the river, as we refer to it, sitting here as the Democratic leader 
on this committee. 

This hearing is called, regrettably. I wish it wasn’t being held. 
I know the Director wishes it wasn’t being held and the Bureau 
does, also, I am sure. 

For a long time, this committee has committed a large number 
of resources, a tremendous amount of effort, on working with the 
Federal Bureau of Investigation (FBI) to try to upgrade the tech- 
nology capability of the agents on the street and the FBI generally, 
not only within the Bureau but as it integrates with the rest of the 
Government, especially on this core issue of how we fight ter- 
rorism. Part of this initiative, of course, has been the famous Tril- 
ogy program, which has had fits and starts, which has involved a 
large number of dollars and in which we have made a serious ef- 
fort. 

The FBI, to begin with, needs to be congratulated. We are 3V2 
years out from 9/11 and we haven’t been attacked, and that is in 
large part because of the excellent work of the FBI and the men 
and women of that agency who commit their lives to making sure 
that we are secure. I congratulate the Bureau for that and the 
American people thank you for it. 

( 1 ) 
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TRILOGY PROGRAM SOFTWARE 

In addition, there have been some successes with the Trilogy pro- 
gram that deserve praise. The bringing online of the hardware was 
done on time and it appears to be well done. 

But the big issue is the software which runs the hardware. Here, 
we have a huge problem. It has been reported that we now have 
independent evaluations and it appears that the Virtual Case File 
(VCF) element, which is essentially the software which would give 
the agents and the FBI the capacity to adequately consolidate and 
track cases from agent to agent, from field office to field office, from 
central command back to field offices, has failed catastrophically. 

And so we have got to address why it failed, first. Second, we 
have to ask the question of who is responsible. I think that is only 
reasonable because there is a large amount of taxpayers’ dollars 
that have produced very little for the taxpayers, over $100 million 
minimum. And then, third, where do we go from here, because this 
is a critical element of having an efficient and effective FBI and es- 
pecially an efficient and effective deterrent to terrorism. So now 
that we have had this very significant failure, how do we get back 
on track and what is the timeframe, what is the cost, and most im- 
portantly, can it be done? 

The Director has kindly agreed to come and testify today. I ap- 
preciate his courtesy in giving us time today on this issue. We are 
going to proceed with trying to find out what is going on and how 
we can fix the problem. We are not too interested in spending a lot 
of time on the history of the blame. We are more interested in fig- 
uring out how we fix the problem. 

Senator Leahy. 

OPENING REMARKS OF SENATOR PATRICK J. LEAHY 

Senator Leahy. Thank you, Mr. Chairman. I agree very much 
with what you had to say. Mr. Chairman, perhaps it is our New 
England upbringing that we get somewhat worried about the 
amount of money that has been put in here and will never be re- 
covered. 

We have an important issue for the FBI in this. I am glad that 
Director Mueller and Inspector General Fine have returned to tes- 
tify. I appreciate the time Director Mueller spent with me yester- 
day, he and Mr. Azmi and others. I made very clear to him my con- 
cern at that time on this and some other subjects. 

POSSIBILITY OE SCRAPING KEY TRILOGY COMPONENTS 

I know Groundhog Day was yesterday, but I think of that movie, 
“Groundhog Day,” and the sense of deja vu the movie had. It is un- 
believable, given the years that have gone by, the advances in tech- 
nology that have marched on in the meantime, that we are here 
today to discuss whether or not to completely scrap a key compo- 
nent of the Trilogy project, the long-anticipated Virtual Case File. 
It has been kind of a train wreck in slow motion, unfortunately, at 
a cost of $170 million to the taxpayers, or a very large part of that. 
We don’t know how much of a cost to the public. 

I don’t want New England reserve to fool anybody to think that 
my reaction getting the initial reports of this was much short of ap- 
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oplectic, this unraveling of the Trilogy project, or as some FBI 
agents have told me privately, the tragedy project. It would bring 
the FBI’s information technology into the 21st century. That 
shouldn’t be rocket science. Most companies have to do that. It 
should be doable. 

This has been a long and tortured effort. Back in 2000, when we 
began discussions about Trilogy as a way to bring the FBI’s anti- 
quated system into the 21st century, we were warned of dire con- 
sequences to our security and our safety if the improvements 
weren’t imminent, if we didn’t give them the money so that it could 
be done right away. Well, we responded. We devoted $581 million 
to the project. 


ASSESSMENTS OF VISUAL CASE FILES 

But time and again, it has fallen victim to escalating costs and 
implementation concerns, mismanagement, and so on. The esti- 
mated December 2003 deadline for completion of it passed unmet. 
The program was then dubbed unusable. We now know that it is 
being tested as the so-called “Virtual Case File (VCF/Light).” 

The $170 million seems to have evaporated. Maybe some of this, 
we can get back from those supplying software and hardware. But 
what bothers me is that a lot of the delays in communications, even 
though we asked in different committees — and I am on the author- 
izing committee as well as the appropriating committee — we never 
seemed — they weren’t communicated to Congress, and it wasn’t be- 
cause Republicans and Democrats alike weren’t asking. We were. 

The FBI has repeatedly pressed for realistic assessments of VCF, 
but getting straight answers from the Department of Justice and 
the FBI have proved so difficult that we finally turned to the Gov- 
ernment Accountability Office (GAO) for an independent assess- 
ment. It is only in the shadow of that impending Office of Inspector 
General (OIG) report that suddenly this comes to light. We have 
a classic example of too many cooks with the unpredictable results. 

The initial contract for VCF was modified 36 times. During this 
period, the FBI had five different chief information officers, I am 
told 10 different project managers. Beyond that shuffling, several 
teams were brought into the process at various times to help set 
requirements, assess deliverables, and manage costs. Even the ef- 
forts of the GAO have been thwarted by changes in personnel and 
trying to get answers. 

Technology changes rapidly, I appreciate that. But the private 
sector has to make these decisions under similar pressures and it 
is not too much to demand the same from the FBI. 

The September 11 attacks did change the FBI’s assessment on 
what is needed. I appreciate that. But 3 years have passed for the 
FBI to regroup. The Congress has responded with the necessary fi- 
nancial resources. 


LESSONS LEARNED 

Now, this has been a very, very expensive lesson learned pro- 
gram. Congress paid for something to be built, not for learning 
what has to be built through trial and error. We have to protect 
the American people. To do this effectively, the FBI has to have 
state-of-the-art technology that works. It is a vital task. Now we 
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are going to have to spend more money to buy what we thought 
we bought. 

But I think that just simply spending money is not going to be 
enough. We can’t just keep throwing money at the problem. I think 
that the FBI has got to stop hiding its problems. The Department 
of Justice has to stop hiding its problems. You know, you have a 
lot of us up here who have been very, very supportive of law en- 
forcement, very supportive of the FBI. I have done this for 30 years 
in both appropriations and authorizations. But, you know, the cam- 
el’s back is broken, and if you think that some of us who have been 
supportive in the past are going to keep on spending money and 
we are not getting answers, or are told all is well when it is not, 
it is just not going to work. 

Mr. Chairman, I agree with you. It is unfortunate we have to be 
having this hearing, but thank goodness we are. 

Senator Gregg. Thank you. 

[The statement follows:] 

Prepared Statement of Senator Patrick J. Leahy 

I commend Chairman Gregg for convening this hearing today. This is an impor- 
tant issue for the FBI and its missions in protecting the country, and I appreciate 
the opportunity to serve as ranking member on this hearing. I am pleased that both 
Director Mueller and Inspector General Fine have returned to testify, I welcome 
them and the other witnesses, and I look forward to their testimony. 

I know that Groundhog Day was actually yesterday, but the subject of today’s 
hearing — problems the FBI is having with its computers — calls to mind the sense 
of deja vu that the film of the same name captured so well. It is unbelievable given 
the years that have gone by and the advances in technology that have marched on 
in the meantime that we are here today to discuss whether or not to completely 
scrap a key component of the FBI’s Trilogy project — the long-anticipated Virtual 
Case File. This program has been a train wreck in slow motion, at a cost of $170 
million to American taxpayers and an unknown cost to public safety. And sadly, 
VCF is but one of many Trilogy problems at the FBI. 

Apoplectic would be too mild a description of my reaction to the unraveling of the 
Trilogy project — or the Tragedy project, as some FBI agents have taken to calling 
it. Bringing the FBI’s information technology into the 21st Century should not be 
rocket science; it is a complicated process, but it is certainly doable. 

The history of the FBI’s efforts to upgrade its information technology has been 
long and tortured. Back in 2000, when we began discussions about Trilogy as a way 
to bring the FBI’s antiquated systems into the 21st Century, we were warned of dire 
consequences to our security and our safety if the improvements were not imminent. 
The picture was bleak. The Bureau had no functional e-mail system at the time, 
and over 13,000 desktop computers that were years old could not run basic software 
packages. Congress responded by devoting $581 million to the effort. 

These deficiencies had real-world consequences, hampering the FBI’s ability to 
share important and time-sensitive information internally and externally with other 
intelligence and law enforcement agencies. In testimony before the Senate Judiciary 
Committee, 9/11 Commissioner Slade Gorton noted: “[IJnformation technology 
problems . . . have hampered the FBI’s ability to know what it knows for years” 
and led to the now infamous incident of the Phoenix memo on terrorists and flight 
schools that never made it to the attention of top officials who should have seen it. 

Time and again, the project has fallen victim to escalating costs, imprecise plan- 
ning, mismanagement, implementation concerns, and delays. The necessary net- 
work, hardware and software upgrades were not delivered in a timely manner. Con- 
sequently, the estimated December 2003 deadline for completion of VCF passed 
unmet. The program was then dubbed “unusable,” and we now know that what is 
being tested is a significantly scaled-down version, a so-called “VCF-lite.” And in 
keeping with the past disappointments and delays, we have recently learned that 
the future of this “lite” version remains in question. 

Congress was led to believe that VCF was progressing on track despite some 
delays and cost overruns on the project. Yet now we hear that VCF may never be 
completed at all. In effect, that means for Congress, for the FBI and, most impor- 
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tantly, for the American teixpayer, this has been $170 million in “vaporware” — wide- 
ly advertised, but never actually available for use. 

There has been no shortage of opportunities for straightforward reporting to the 
oversight committees of Congress as things began to come off the tracks, including 
numerous hearings, punctuated by several damaging reports from OIG, the Govern- 
ment Accountability Office, and the National Research Council. These delays and 
disappointments were never communicated to Congress, and it is not because Con- 
gress failed to ask. The FBI was repeatedly pressed for realistic assessments of 
VCF. But getting straight answers from the Justice Department and the FBI proved 
so difficult that Congress finally turned to the Government Accountability Office for 
an independent assessment. It was only in the shadow of an impending OIG report 
that the reality of the situation has come to light. 

Director Mueller testified before the Judiciary Committee last May and was spe- 
cifically asked about the status of VCF. He testified then that “we are on track to 
deliver elements of virtual case file capabilities by the end of this year. We are in 
negotiations with our contractor on finishing out that last part of the Trilogy 
project . . . But I do believe that when we are concluded this year, we will have 
the foundation for cutting-edge technology for an organization our size.” 

What was not presented in this hearing was any acknowledgement or even any 
hint that progress had halted and the project was, in fact, falling apart. This was 
an opportunity for Director Mueller to show some accountability and be upfront 
with Congress about the problems with the project. The FBI missed another oppor- 
tunity to come clean three months later when the Committee convened a hearing 
on the 9/11 Commission’s recommendations. 

It appears the FBI bears the brunt of the responsibility for this derailment; a clas- 
sic example of too many cooks, with the predictable results. The initial contract for 
VCF was modified 36 times. During this period, the FBI had five different Chief In- 
formation Officers and, reportedly, 10 different project managers. Beyond all that 
shuffling at the top, several teams were brought into the process at various times 
to help set requirements, assess deliverables and manage costs. Even the recent ef- 
forts of the GAO to audit the project have been thwarted by repeated changes in 
the personnel responding to auditors’ inquiries. 

It is not clear to me even yet what the FBI truly knew and whether the Bureau 
articulated what it needed, though initial reports suggest the FBI made an art form 
of redefining and changing its requirements. The project’s contractor. Science Appli- 
cations International Corporation, has said it received changes on almost a daily 
basis — some small, but many, significant. Unbelievably, the OIG reports that the 
process for defining the requirements and baselines for the VCF continues to this 
day. I look forward to hearing from Inspector General Fine on this matter. The Tril- 
ogy project is reminiscent of other FBI technology failures where the Bureau has 
ambitiously tried to build the latest and greatest without properly assessing its 
needs. The FBI custom-built the Carnivore system on the basis that it was “far bet- 
ter” than any commercial product, but after very little use, recently scrapped it for 
an undisclosed commercial product. 

Technology changes rapidly, and I appreciate the FBI’s efforts to keep pace. But 
the private sector has had to make these hard decisions with similar pressures, and 
it is not unreasonable to demand as much from the FBI. The September 11th at- 
tacks did change the FBI’s assessment of what it needed. But three years have 
passed for the Bureau to regroup, and in that time Congress has responded with 
the necessary financial resources to assist the Bureau in adapting in these tasks. 
This has been an outrageously expensive lessons-learned training program. Con- 
gress paid for something to be built, not for learning about what to build through 
trial and error. 

I am aware of the concerns that have also been raised about the performance of 
SAIC, the project’s contractor, and I do expect SAIC to account for any failures in 
its work product. 

Our highest priority must be to protect the American people. To do its job effec- 
tively, the FBI must have state-of-the-art technology that works. This is a vital task, 
and now Congress will have to provide still more funding to get the job done. But 
throwing money at this chronic problem alone will not fix it. The FBI must stop hid- 
ing its problems and begin confronting them. The FBI needs to engage in a full 
working partnership with the authorizing and appropriations committees to which 
the Bureau is accountable to for programs like this. Doing that will better protect 
the public, conserve tax dollars, and save everyone’s time. 

The camel’s back is broken. For a course correction to succeed, there must be a 
true accounting, and it is going to start today. We want to hear what went wrong, 
who was responsible, and how we are going to move forward. 
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Senator Gregg. Traditionally, we haven’t had opening state- 
ments heyond the chairman and the ranking member, but obvi- 
ously, participation today is by folks who are really interested in 
this and I didn’t know whether you wanted to make a statement. 

STATEMENT OE SENATOR BARBARA A. MIKULSKI 

Senator Mikulski. Just very briefly, Mr. Chairman. First of all, 
I want to thank you for holding this hearing. You have always 
stood sentry over getting taxpayers’ value for taxpayers’ dollar, and 
you were the first to hold public hearings on the issue of terrorism, 
so we thank you for your leadership. 

Also to Senator Leahy, in the absence of a permanent Chair of 
CJS, we thank you for filling in. It is also very possible that if the 
draconian restructuring program of the House would ever go 
through, I might Chair this subcommittee, which 

Senator Gregg. Or be ranking. 

Senator Mikulski. Or just be ranking. 

Oh, no, that is another restructuring. 

Senator Leahy. I wasn’t going to say a word. 

Senator Mikulski. I am sorry. 

Senator Leahy. I am just staying out of this one. 

Senator Mikulski. I am sorry. I was so excited. That was regime 
change, not restructuring. 

But I also wanted to be here as a member of the committee. I 
am a member of the Intelligence Committee. We went through the 
9/11 inquiry and we were absolutely very clear that our FBI needed 
to modernize itself. We are proud of the FBI and we are proud of 
the fact that we have asked them to retool their mission, retool 
their people, and retool their technology. And now, as we have 
moved forward to the reform necessary for both intelligence and 
FBI, I think the Director is working very hard on this retooling of 
the mission. The people that he has hired have helped him do this. 
Now we have to make sure that we have the right technology to 
do this. 


TECHNOLOGY PROGRAMS GOING BUST 

Right after 9/11, we found out that the FBI had 13,000 desktop 
computers that were outdated and dysfunctional. We also saw that 
that whole idea of watch lists talking to each other, offices talking 
to each other, and so on was outdated. We have got to get this back 
on track. 

As someone who has looked at these big technology programs, 
whether it was in transportation, whether it was out of the VA/ 
HUD Subcommittee, they have always been a bust. I think maybe 
we have to reexamine that rather than inventing things, that we 
need to look at how to buy things off the shelf, how we can move 
quicker, faster, cheaper, and save a lot of heartache, a lot of heart- 
burn, and a lot of taxpayers’ dollars. 

But I know today is the day for getting the FBI and its financial 
and computer programs back on track and I look forward to work- 
ing with you in any capacity in which I might find myself. 

Senator Gregg. I look forward to that, also. 

Senator Mikulski. I am ready to retool if I have to. 
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Senator Gregg. Mr. Director, we are ready to hear your thoughts 
and explanations. 

STATEMENT OF HON. ROBERT S. MUELLER, III, DIRECTOR, FEDERAL 
BUREAU OF INVESTIGATION, DEPARTMENT OF JUSTICE 

ACCOMPANIED BY ZALMAI AZMI, CHIEF INFORMATION OFFICER, FED- 
ERAL BUREAU OF INVESTIGATION, DEPARTMENT OF JUSTICE 

OPENING STATEMENT OF DIRECTOR MUELLER 

Mr. Mueller. Thank you, Mr. Chairman, Senator Mikulski, Sen- 
ator Stevens. I do want to thank you, believe it or not, for the op- 
portunity to he here today to discuss this issue because it is impor- 
tant. It is important to the Federal Bureau of Investigation (FBI), 
it is important to the country, and I know it is important to the 
Congress. 

I do want to spend some time discussing the questions that you, 
Mr. Chairman, have raised. As all of us know, the Virtual Case 
File is a case management system constituting the third prong of 
the FBI’s mission technology program and is known as Trilogy. It 
was first developed in 2001. 

COMPLETED PHASES OF TRILOGY 

I want to point out at the outset that the first two phases of Tril- 
ogy have been successfully completed and, as the chairman pointed 
out, have been deployed and have greatly enhanced our informa- 
tion technology capabilities. We now have deployed a high-speed 
network enabling our FBI offices around the country and around 
the world to share data, including audio, video, and image files. 
Our new IT infrastructure also provides for secure communications 
with our intelligence community partners. 

We have replaced those outdated computers with more than 
30,000 new desktop computers with modern software applications, 
and we have replaced nearly 4,000 printers. We have 1,600 scan- 
ners, 465 servers, and as important, 1,400 routers that have been 
installed. 

As a result of the implementation of the first two prongs of Tril- 
ogy, FBI personnel can now utilize a uniform suite of software that 
enables our agents and our support to share information quickly, 
reliably, and securely. 

These efforts have also provided a foundation for a number of 
new capabilities to support the FBI’s counterterrorism mission. I 
will point out at the outset that after September 11, while Trilogy 
and bringing Trilogy home was tremendously important, it also at 
that time was critically important to us to take our 
counterterrorism information throughout the Bureau, information 
from elsewhere on counterterrorism, and place that information in 
an updated investigative data warehouse. We now have that infor- 
mation, that investigative data warehouse, that has that informa- 
tion and provides to special agents, intelligence analysts, and mem- 
bers of joint terrorism task forces a single access point to more 
than 47 sources of counterterrorism data that was only in the past 
available through separate stovepiped systems. 

We have new analytical tools used across multiple data sources, 
providing a more complete view of the information possessed by the 
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Bureau. Users can now search up to 100 million pages of inter- 
national terrorism-related documents and other structured records, 
such as addresses and phone numbers, in seconds. They can also 
search rapidly for pictures of known terrorists and match or com- 
pare the pictures with other individuals in minutes rather than 
days. 


CRITICAL IT IMPROVEMENTS 

Other critical IT improvements have enabled the FBI to proceed 
with unprecedented connectivity with our partners in the intel- 
ligence and law enforcement communities. The SCION network 
gives FBI personnel the ability to electronically receive, dissemi- 
nate, and share compartmented sources of intelligence information 
amongst our various operating divisions and with the intelligence 
community. 

But despite these significant improvements, the Virtual Case 
File, which is a case management application for improving effi- 
ciency and records management, is not yet available to our per- 
sonnel. I can tell you, Mr. Chairman, as I have expressed in private 
to yourself and Senator Leahy, there is no one who is more frus- 
trated, no one who is more disappointed than I at the delays we 
have encountered in deploying VCF. I do believe, however, it is im- 
portant to the American people to understand what the failure to 
deliver VCF means, and what it does not mean, to the FBI agent 
on the street. 


TECHNOLOGY FOR STREET AGENTS 

I want to point out that the FBI agent on the street has the 
state-of-the-art technology when it comes to surveillance. Without 
getting into sensitive and classified information, I can assure you 
that our ability to intercept and decipher communications and to 
otherwise monitor criminal activity and gather intelligence is 
among the best in the world. The FBI agent on the street is able 
to communicate and share data securely, whether by telephone, 
computer, or teleconference with our partners, not only in the FBI 
but also in the law enforcement and intelligence communities in 
the United States and around the world. 

What the agent on the street does not have is a user-friendly for- 
mat for inputting investigative and intelligence information into 
his or her computer. Instead, the agent faces a cumbersome, time- 
consuming process of preparing a paper record of that information, 
seeking the necessary approvals, and then uploading that docu- 
ment into an existing database. If the agents had the Virtual Case 
File capabilities we had envisioned, they could directly input infor- 
mation into their computers, receive electronic approvals, and with 
the push of a button upload information into the database where 
it would be immediately available to others who need to access it, 
whether it be an agent, an analyst, or other Federal employees and 
State and local officials. 

And by saying this, Mr. Chairman, I do not mean to say that this 
does not affect our capacity to protect the United States. To the ex- 
tent that we are delayed, to the extent that we do not have this 
Virtual Case File, we are not as effective or efficient as we should 
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be in protecting the people of the United States, whether it be from 
terrorism or criminals within the country. 

VIRTUAL CASE FILE ANSWERS 

Mr. Chairman, this afternoon, I would like to take this oppor- 
tunity to answer the three basic questions about Virtual Case File 
which you elucidated in your opening statement. First of all, what 
went wrong? Second, who is responsible for what went wrong? And 
third, where do we go from here? 

What went wrong? The development of the VCF application 
started with a relatively simple concept that the FBI needed a 
modern case management system. As the FBI’s mission evolved, 
particularly over the past 3 years, so did our technological needs. 
And as a result of these changes and other issues, the FBI faced 
obstacles in a number of key areas relating to the VCF program. 

We did not have a complete set of defined VCF requirements 
when the original contract was signed in June 2001, and we did 
not have a finalized set until the summer of 2002. 

The contract which we entered into was based on hours worked, 
a cost plus award fee, and we now know that these types of con- 
tracts are difficult to manage. 

But from our perspective, we also lacked skill sets in our per- 
sonnel, such as qualified software engineering, program manage- 
ment, and contract management. 

We underestimated the complexity of interfacing with our legacy 
systems, of addressing our security needs, and of establishing an 
enterprise architecture. 

Recognizing many of those internal limitations originally, we did 
decide to outsource the development of VCF, including contract 
management and technology development. The contractor respon- 
sible for delivering the user applications component, including 
VCF, was Science Applications International Corporation (SAIC). I 
know you are to hear from them today, as well. 

Following the establishment of the solid requirements in Novem- 
ber 2002, the original target date for completing VCF was Decem- 
ber 2003. I personally received a demonstration of the VCF soft- 
ware in November 2003, and was impressed by what I saw at that 
time. I anticipated that we would be moving forward expeditiously 
to the installing of that VCF on our agents’ support computers in 
the relatively near future once we had upgraded all of our com- 
puters from a Windows 98 operating system to a Windows 2000 op- 
erating system. I, at that time, believed that we were on the right 
track to deliver that which our employees were seeking. 

When SAIC delivered the first product in December 2003, we im- 
mediately identified a number of deficiencies, 17 at the outset. That 
soon cascaded to 59 and ultimately to 400 problems with that soft- 
ware. In April 2004, we provided SAIC with a document outlining 
the corrections we felt were needed and SAIC ultimately agreed to 
remedy the deficiencies and deliver full functionality, but only at 
a cost, an additional $56 million, and a timetable, an additional 
year, which at that time we had problems with. 
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TWO-TRACK VISUAL CASE FILE PLAN ADOPTION 

So in June 2004, I decided to adopt a new two-track plan for 
VCF, an initial operating capability, or IOC, and a full operating 
capability, which is denominated as FOC. My goal with the IOC 
was to identify and utilize some portion of the product developed 
by SAIC since the fully functional case management system as we 
had anticipated had not been delivered. The portion of Virtual Case 
File currently being piloted is the automated workflow process. 
Last month, several hundred employees in the New Orleans field 
office began using the system as their document routing system 
and will continue to do so through the end of March. 

The purpose of this pilot is to test drive the workflow concept, 
validate the human-computer interface, create an electronic inter- 
face to our legacy systems, access the network performance, and 
develop and deliver an enterprise-level training curriculum. The 
IOC, the initial operating capability, is on track to accomplish 
these objectives. 

As part of two-track plans, the FBI contracted with multiple 
independent vendors to perform the following tasks: Examine the 
Virtual Case File application delivered by SAIC in December 2003, 
to determine if this software, as designed, would meet the FBI’s 
operational, security, and performance requirements. Aerospace 
Corporation was tasked to determine if the Virtual Case File appli- 
cation is scalable and can be maintained and enhanced easily. 

They were also asked to examine the current technologies and 
vendors as well as available commercial off-the-shelf or COTS, 
products. They were also tasked to look at those products designed 
for other agencies to determine the best combination to meet the 
FBI’s needs. This effort was conducted jointly, not only with our- 
selves and the Department of Justice, but also with the Depart- 
ment of Homeland Security, to ensure our case management efforts 
would be interoperable. In many ways, as several of you have 
pointed out, the pace of technological innovation and the need for 
information sharing has overtaken our original vision for Virtual 
Case File and there are now products to suit our purposes that did 
not exist when Trilogy was first envisioned. 

We have also asked a different contractor to review and revali- 
date our users’ requirements because the mission of the FBI has 
evolved and there are new requirements for information and intel- 
ligence sharing among different entities. 

Last week, we received the final version of the Aerospace report 
and provided copies to this subcommittee and to the Office of In- 
spector General at the Department of Justice. 

RESPONSIBILITY FOR WHAT WENT WRONG 

Question number two, who is responsible for what went wrong? 
Mr. Chairman, I am responsible, at least in part, for some of the 
setbacks experienced with Trilogy and Virtual Case File. I agree 
with the OIG’s findings that FBI management did not exercise ade- 
quate control over the Trilogy project and its evolution in the early 
years of the project. 

Let me also add that I agree with the OIG’s finding that with 
the new organizational structure and authority given to the Chief 
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Information Officer (CIO), Zal Azmi, in July 2004, project manage- 
ment has now been given the attention that was needed through- 
out the Trilogy project. Zal Azmi is here with me today. He started 
with me as a special advisor on technology issues when I first saw 
problems in the fall of 2003. He became the Chief Information Offi- 
cer in spring of last year, and through his leadership, the FBI has 
implemented a coordinated strategic approach to information tech- 
nology. My prepared statement outlines a number of the steps that 
Mr. Azmi has taken as CIO and some of the accomplishments of 
him and the people with whom he works. 

I also will say, and I think it is shared in the testimony from 
SAIC, that in addition to our shortcomings in overseeing the Tril- 
ogy project, the contractor also bears some responsibility. As dis- 
cussed above, we retained a not-for-profit federally funded con- 
tractor, Aerospace Corporation, to conduct an independent 
verification and validation review of the Virtual Case File, the VCF 
software as delivered by SAIC in December 2003. 

Aerospace in its report concluded that, and I quote, “lack of effec- 
tive engineering discipline has led to inadequate specification, de- 
sign, and development of VCF.” In the course of their review. Aero- 
space could find no assurance that the requirements were satisfied 
nor that the architecture concept of operations and requirements 
were correct and complete. When we received this report recently, 
we were indeed disappointed. 

VIRTUAL CASE FILE FUNDING 

With regard to the funding of Virtual Case File, this committee 
has been supportive of our efforts and has generously provided the 
funding we have needed to overcome obstacles and attempt to move 
forward. Mr. Chairman, you and other members are undoubtedly 
concerned, as am I, about losses we have incurred as well as future 
investments we will need to make in Virtual Case File. 

We have invested approximately $170 million in VCF to date. It 
is my understanding that our vendors have delivered services and 
reusable equipment worth $53.3 million and that we have $12.2 
million in unspent obligations on our VCF contracts. This results 
in a loss of approximately $104.5 million. I do not take that lightly. 
It is $104.5 million that we will not have to spend on other things. 
It is $104.5 million of the taxpayers’ dollars and I am tremendously 
troubled by that and that is an understatement. I am disheartened 
by this result, but remain confident in our ability now to deliver 
a case management system to our employees’ desktops in the fu- 
ture. 


WHERE DO WE GO FROM HERE 

Last question, where do we go from here? The development and 
deployment of an investigative case management system remains 
the top priority of the Office of the Chief Information Officer. Some 
components of VCF that have been developed will be incorporated 
into the long-term solution. We will leverage the permanent inter- 
face that has been established with our legacy data systems. We 
will assess the impact of an automated workflow system on a field 
office and headquarters structure as well as the performance of a 
case management system on the new Trilogy network, during, and 
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at the end, of the pilot testing period. We will take with us a num- 
ber of valuable lessons learned in contract management, project 
management, policies and procedures, modular development and 
deployment, data security, and records management. 

Not surprisingly, the pace of technology has overtaken the devel- 
opment of unique software applications for the Bureau and we may 
turn to commercial off-the-shelf, or COTS-based products, to give 
us that which we had envisioned in Virtual Case File. We are cur- 
rently reviewing the Aerospace reports which recommend that we 
discard VCF and start over with a COTS-based product and which 
provide their evaluation of COTS products as well as products in 
use by other Government agencies. As we review these reports, we 
will continue to consult with industry leaders to ensure that we de- 
velop a sound long-term plan for our IT needs. 

We will move forward with a phased, and I emphasize a phased, 
development and deployment plan as recommended by the Na- 
tional Academy of Sciences. Every phase will provide a set of serv- 
ices that the FBI workforce needs and which was part of the origi- 
nal VCF plan. 

I cannot at this time estimate when this will occur, nor can I de- 
termine right now what we will need in terms of additional fund- 
ing. I will tell you that we will work closely with this committee 
and other committees of Congress to develop the future for a Vir- 
tual Case File, and with the work of Mr. Azmi and the people he 
has brought in, with input from persons outside the Bureau, I am 
confident that we, in a phased way, can replicate that which we 
had envisioned in 2000 and 2001 as being a part of Virtual Case 
File. 


AEROSPACE CORPORATION SELECTION 

Mr. Chairman, before I conclude, let me say that I have reviewed 
the testimony of other witnesses and there are two questions that 
I would anticipate and would like to answer at the outset. 

The first question is, how did we select Aerospace Corporation to 
conduct the independent verification and validation review, and I 
am going to pass that over to Mr. Azmi. 

I am going to start on the second question, and that is why did 
we limit Aerospace’s review to the December 2003 delivery of Vir- 
tual Case File and not include that which was produced in Decem- 
ber 2004 and that which we are testing now. 

I will tell you, last spring, in 2004, after we saw the problems 
we had in the version that was provided in December 2003, we en- 
tered into negotiations with SAIC, and at the end of those negotia- 
tions it was clear from their leaders that we would have to invest 
another $56 million and an additional year of time to complete the 
project as we had anticipated with SAIC. At that time, in consulta- 
tion with Mr. Azmi, I felt we needed an independent review of the 
work that had been produced by SAIC and that is the version that 
we had to review at that time. I am comfortable in having Aero- 
space or anyone else review the initial operating capacity that is 
currently being tested in New Orleans and here at the FBI head- 
quarters. 

Mr. Azmi may want to provide more input into why we asked 
Aerospace to review the December 2003 delivery, and I would also 
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ask him to answer the question as to why we selected Aerospace, 
because I believe that is when it would be forthcoming, and then 
I will close. 

Mr. Azmi. Mr. Chairman and members of the committee, thank 
you for the opportunity to appear here today and respond to your 
questions. 

On the question, why we selected Aerospace to conduct an eval- 
uation of the VCF delivery of 2003, it was mainly a recommenda- 
tion made by the Director of the Science Board. When I arrived in 
November 2003, I realized that the Director already had a number 
of boards and advisors that were actually providing input to the fu- 
ture of the information technology within the Bureau. I met with 
the Science Board — the members are former CIOs, technologists 
from both the Government and private sector — and I presented the 
dilemma that we were facing with VCF. 

The software was delivered with 17 deficiencies. We decomposed 
those 17 deficiencies to 59. Later on, we found 400 problems with 
the software, and that was the recommendation of the Board, that 
we conduct an independent evaluation. 

We had selected three sources of evaluators. Aerospace was se- 
lected because it was a federally funded organization, a nonprofit 
organization. It had worked with the Department of Defense (DOD) 
and the intelligence community for more than four decades. They 
were also capable of providing in-depth software engineering re- 
view that we needed. For those reasons, we selected Aerospace to 
conduct an independent evaluation of the VCF software. Thank 
you, sir. 

Mr. Mueller. Anything to add on why we selected the December 
2003 version to be evaluated? 

Mr. Azmi. I can add only one point. When I arrived, I looked at 
the contract and the contract stated specifically that SAIC will de- 
ploy a working version of VCF by December 17, 2003. When I 
looked at all of the capabilities of VCF, what should have been de- 
livered and what was delivered, we decided if we are going to in- 
vest in the software for the future of the FBI, if we will have to 
stay with this software, we need to understand what the software 
will provide to us, and that is one of the reasons why we selected 
to evaluate that software that was promised to the Bureau from 
the outset of support of this contract. 

PREPARED STATEMENT 

Mr. Mueller. So in closing, those hopefully answer the ques- 
tions that would have been asked. I want to thank the sub- 
committee, you in particular, Mr. Chairman, for your support 
throughout this endeavor, your patience, understanding your frus- 
tration. Mr. Azmi and I are happy to respond to any questions that 
the subcommittee may have. 

Senator Gregg. Thank you, Mr. Director, and thank you for your 
forthrightness. 

[The statement follows:] 

Prepared Statement of Robert S. Mueller, III 

Good afternoon, Mr. Chairman, Senator Leahy, and Members of the Committee. 
Thank you for the opportunity to appear this afternoon and address concerns relat- 
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ing to the FBI’s Virtual Case File system, or VCF. As you know, VCF is a case man- 
agement system constituting the third prong of the FBI’s information technology 
program known as Trilogy. The first two phases of Trilogy have been successfully 
completed and deployed, and have greatly enhanced our Information Technology 
(IT) capabilities. 

— We have deployed a high-speed, secure network, enabling personnel in FBI of- 
fices around the country and around the world to share data, including audio, 
video and image files. Our new IT infrastructure also provides for secure com- 
munications with our Intelligence Community partners. 

— We have also replaced outdated hardware with more than 30,000 new desktop 
computers with modern software applications, nearly 4,000 new printers, 1,600 
scanners, 465 servers, and 1,400 routers. 

As a result of the implementation of two major prongs of the Trilogy initiative, 
FBI personnel can now utilize a uniform suite of software that enables them to 
share information quickly, reliably, and securely. These efforts have also provided 
a foundation for a number of new capabilities to support the FBI’s counterterrorism 
mission. The new capabilities include: 

— The FBI’s Investigative Data Warehouse (IDW) now provides Special Agents, 
Intelligence Analysts, and members of Joint Terrorism Task Forces (JTTFs) 
with a single access point to more than 47 sources of counterterrorism data, in- 
cluding information from FBI files, other government agency data, and open 
source news feeds, that were previously available only through separate, stove- 
piped systems. 

— New analytical tools are used across multiple data sources providing a more 
complete view of the information possessed by the Bureau. Users can search up 
to 100 million pages of international terrorism-related documents and other 
structured records such as addresses and phone numbers in seconds. They can 
also search rapidly for pictures of known terrorists and match or compare the 
pictures with other individuals in minutes rather than days. Coupled with so- 
phisticated state-of-the-art search tools, the IDW enhances the FBI’s ability to 
identify relationships across cases quickly and easily. 

— Other critical IT improvements have enabled the FBI to proceed with unprece- 
dented connectivity with our partners in the Intelligence and Law Enforcement 
Communities. The Top Secret/Sensitive Compartmented Information Oper- 
ational Network (SCION) gives FBI personnel the ability to electronically re- 
ceive, disseminate, and share compartmented sources of intelligence information 
among the FBI’s counterterrorism and counterintelligence operations and with 
the Intelligence Community. SCION also provides for video teleconferencing at 
the TOP SECRET level. 

Despite these significant improvements, the Virtual Case File — a case manage- 
ment application for improving efficiency and records management — is not yet avail- 
able to our personnel. Mr. Chairman, no one is more frustrated and disappointed 
than I at the delays we have encountered in deploying VCF. But I believe it is im- 
portant for the American people to understand what the failure to deliver VCF 
means — and what it doesn’t mean — to the FBI Agent on the street. 

The FBI Agent on the street has state-of-the-art technology when it comes to sur- 
veillance. Without getting into sensitive and classified information, I can assure you 
that our ability to intercept and decipher communications and to otherwise monitor 
criminal activity and gather intelligence is among the best in the world. The FBI 
Agent on the street is able to communicate and share data securely, whether by 
telephone, computer, or teleconference with our partners, not only in the FBI, but 
also in the law enforcement and intelligence communities, in the United States and 
around the world. The Agent on the street is able to access FBI documents electroni- 
cally on our existing computer systems and to search those documents using mul- 
tiple search technologies. 

What the Agent on the street does not have is a user-friendly format for inputting 
investigative and intelligence information into his or her computer. Instead, the 
Agent faces a cumbersome, time-consuming process of preparing a paper record of 
that information, seeking the necessary approvals, then uploading the document 
into an existing database. If Agents had the VCF capabilities we envisioned, they 
could directly input information into their computers, receive electronic approvals, 
and, with the push of a button, upload information into the database where it would 
be immediately available to others who need access to it — Agents, analysts, other 
federal employees, and state and local officials. 

I want to emphasize, however, that although VCF would enable us to do our jobs 
more efficiently, the absence of VCF does not prevent us from fulfilling our 
counterterrorism, intelligence and law enforcement missions. Again, VCF is not a 
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database or an analytical tool used to connect the dots — it is a case management 
system that will make it easier for Agents to input and share the dots. 

Having said that, Mr. Chairman, I thank you for your longstanding interest in 
the VCF program and your commitment to hold a public hearing to examine the 
setbacks which have plagued this program. This afternoon, I would like to take the 
opportunity to answer three basic questions about VCF: (1) What went wrong? (2) 
\^o is responsible for what went wrong? and (3) Where do we go from here? 

What Went Wrong? 

The development of the VCF application started with a very simple concept — the 
FBI’s need for a modern case management system. As the FBI’s mission evolved 
over the past several years, so did our technological needs. As a result of these 
changes and other issues, the FBI faced obstacles in a number of key areas relating 
to the VCF program. 

— We did not have a complete set of defined VCF requirements when the original 
contract was signed in June 2001. 

— The contract was based on hours worked — cost plus an award fee. We now know 
these types of contracts are difficult to manage. Although the requirements 
were solidified in November 2002, the contract remained a cost-plus-award-fee 
contract. 

— We lacked skill sets in our personnel such as qualified software engineering, 
program management, and contract management. We also experienced a high 
turnover in Trilogy program managers and Chief Information Officers. 

— We underestimated the complexity of interfacing with our legacy system, of ad- 
dressing our security needs, and of establishing an enterprise architecture. 

We will continue to confront these lessons moving forward. 

Recognizing our internal limitations, we decided to outsource the development of 
VCF, including contract management and technology development. The contractor 
responsible for delivering the user applications component, including VCF, is the 
Science Applications International Corporation, or SAIC. 

Following the establishment of solid requirements in November 2002, the original 
target date for completing VCF was December 2003. I personally received a dem- 
onstration of the VCF software in November 2003 and was impressed with what I 
saw. I believed that we were on the right track to deliver to our employees’ desktops 
the case management system we were seeking. However, when SAIC delivered the 
product to us in December 2003, we immediately identified a number of deficiencies 
in VCF that made it unusable. Upon further examination, we discovered nearly 400 
problems with the software and, in April 2004, provided SAIC with a document out- 
lining the corrections needed. SAIC ultimately agreed to remedy the deficiencies and 
deliver full functionality but only at a cost — an additional $56 million — and a time- 
table — an additional year — which were unacceptable to the FBI. 

In June 2004, I decided to adopt a new two-track plan for VCF: an Initial Oper- 
ating Capability, or IOC, and a Full Operating Capability, or FOC. My goal with 
the IOC was to identify and utilize some portion of the product developed by SAIC, 
since the fully functional case management system had not been delivered. The por- 
tion of VCF currently being piloted in the lOC is the automated workflow process. 
Last month, several hundred employees in the New Orleans field office began using 
the system as their document routing system and will continue to do so through the 
end of March. The purpose of the pilot is to: test drive the workflow concept; vali- 
date the human/machine interface; create an electronic interface to our legacy sys- 
tem, the Automated Case Support System, or ACS; assess network performance; 
and develop and deliver an enterprise level training curriculum. 

The IOC is on track to accomplish these objectives. 

As part of Track Two, the FBI contracted with multiple independent vendors to 
perform the following tasks: 

— Examine the VCF application delivered by SAIC in December 2003 to determine 
if the software as designed will meet the FBI’s operational, security, and per- 
formance requirements. The contractor. Aerospace Corporation, was also tasked 
to determine if the VCF application is scalable and can be maintained and en- 
hanced easily. 

— Examine the current technologies and vendors, as well as available Commercial 
Off-The-Shelf, or COTS, software applications and those designed for other 
agencies, to determine the best combination to meet the FBI’s needs. This effort 
was conducted jointly with the Department of Homeland Security to ensure our 
case management efforts would be interoperable. In many ways, the pace of 
technological innovation and the need for information sharing has overtaken 
our original vision for VCF and there are now products to suit our purposes 
that did not exist when Trilogy began. 
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— We have also asked a different contractor to review and revalidate our users’ 
requirements because the mission of the FBI has evolved and there are new re- 
quirements for information and intelligence sharing among different entities. 

Last week, we received the final version of the Aerospace report and provided cop- 
ies to the Committee and to the Office of the Inspector General at the Department 
of Justice. 

Who is Responsible for What Went Wrong? 

Mr. Chairman, I am responsible, at least in part, for some of the setbacks experi- 
enced with Trilogy and VCF. I agree with the OIG’s finding that “FBI management 
did not exercise adequate control over the Trilogy project and its evolution in the 
early years of the project.” Let me also add that I agree with the OIG’s finding that 
“with the new organizational structure and authority given to the CIO in July 2004, 
project management has been given the attention that was needed throughout the 
Trilogy project.” Mr. Chairman, I will address that new structure and its accom- 
plishments later in my statement. 

In addition to our shortcomings in overseeing this project, however, the contractor 
responsible for VCF bears some responsibility. As discussed above, the FBI retained 
a not-for-profit, federally funded contractor. Aerospace Corporation, to conduct an 
independent verification and validation review of the VCF software as delivered by 
SAIC in December 2003. We asked Aerospace to provide responses to the following 
three questions: 

— 1. Did SAIC meet the stated requirements? 

— 2. Did SAIC develop a complete and correct Concept of Operations, System Ar- 
chitecture, and System Requirements? 

— 3. What should the FBI do with the VCF software as delivered in December 
2003? 

Aerospace concluded that “lack of effective engineering discipline has led to inad- 
equate specification, design and development of VCF.” In the course of their review. 
Aerospace could “find no assurance” that the requirements were satisfied, nor that 
the architecture. Concept of Operations, and requirements were correct and com- 
plete. Needless to say, Mr. Chairman, after three and a half years, this was dis- 
appointing news. 

With regard to the funding of VCF, this Committee has been supportive of our 
efforts and has generously provided the funding we have needed to overcome obsta- 
cles and attempt to move forward. Mr. Chairman, you and the other members are 
undoubtedly concerned — as am I — about losses we have incurred, as well as future 
investments we will need to make, in VCF. We have invested approximately $170 
million in VCF to date. It is my understanding our vendors have delivered services 
and reusable equipment worth $53.3 million and that we have $12.2 million in 
unspent obligations on our VCF contracts. This results in a loss of $104.5 million. 
I am disheartened by this result but remain confident in our ability to deliver a case 
management system to our employees’ desktops in the future. 

Where Do We Go from Here? 

VCF 

The development and deployment of an investigative case management system re- 
mains the top priority of the Office of the CIO. Some components of VCF that have 
been developed will be incorporated into the long-term solution. We will 

— Leverage the permanent interface that has been established with our legacy 
data systems. 

— Assess the impact of an automated workflow system on a field office and Head- 
quarters structure, as well as the performance of a case management system 
on the new Trilogy network, during and at the end of the pilot testing; and, 

— Take with us a number of valuable “lessons learned” in contract management, 
project management, policies and procedures, modular development and deploy- 
ment, data security, and records management requirements. 

Not surprisingly, the pace of technology has overtaken the development of unique 
software applications for the FBI, and we may turn to Commercial Off-The-Shelf, 
or COTS-based, products. We are currently reviewing the Aerospace reports which 
recommend that we discard VCF and start over with COTS-based products, and 
which provide their evaluation of COTS products as well as products in use by other 
government agencies. As we do so, we will continue to consult with industry leaders 
to ensure that we develop a sound, long-term plan for our IT needs. 

We will move forward with a phased development and deployment plan as rec- 
ommended by the National Academy of Sciences and required by Federal informa- 
tion resource management policy. An incremental approach ensures development 
and acquisition of the best available products on the market. Every phase will pro- 
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vide a set of services that the FBI workforce needs and which was part of the origi- 
nal VCF plan. I cannot at this time estimate when this will occur nor how much 
in additional funding we will need to invest to get there. 

We will also give consideration to a Service Oriented Architecture (SOA), as rec- 
ommended in the Aerospace report. The concept behind an SOA solution is to stand- 
ardize enterprise services — such as searching, reporting, and analyzing data — so 
that different groups of users can reuse similar services to access dissimilar data 
sets throughout the enterprise — such as our legacy systems of ACS, III, and our 
Telephone Application. It appears that an SOA approach could provide a flexible so- 
lution to the inflexible systems currently existing within the FBI and would help 
us successfully implement a final product. 

FBI Information Technology 

With me today is Zal Azmi, who joined the FBI in November 2003 as the Chief 
Information Officer. Through his leadership, the FBI has implemented a coordi- 
nated, strategic approach to information technology. 

— Strategic Plan. — In December 2004, we completed our first release of the Stra- 
tegic IT Plan which maps out how IT will support the FBI’s and DOJ’s Strategic 
Plan and mission goals over the next five years. All IT projects are required to 
be consistent with the FBI’s and DOJ’s Strategic Plans. 

— Enterprise Architecture. — We established our baseline Enterprise Architecture 
(EA) in 2004 and are in the process of developing our target EA. We have cre- 
ated an IT Master Systems List identifying all of the IT systems, applications, 
networks and databases in the FBI and DOJ. All IT projects in the future will 
be required to be consistent with the FBI’s and DOJ’s EA. 

— Process Improvement. — Our Life Cycle Management Directive (LCMD), which 
governs how IT projects are managed from “cradle to grave,” is now consistent 
with industry best practices and Federal government information resource man- 
agement policies. All IT Projects and Programs are required to pass through rig- 
orous project and executive level control “gate” reviews for each stage, from in- 
ception through disposal. There are 7 gates, 9 phases, and 14 key supporting 
processes in the LCMD. These reviews are the mechanisms for management 
control and direction, decision-making, coordination, and confirmation of suc- 
cessful performance. 

— Portfolio Management Program. — This program focuses on performance assess- 
ments of IT investments in the operations and maintenance (O&M) phase of 
their life cycle. Since the majority of our IT investments currently reside in the 
O&M phase, these assessments help senior management make more informed 
decisions about IT investments, in terms of both personnel and money. Portfolio 
Management recommendations are focused on those investments that should be 
leveraged, replaced, outsourced or retired. 

— Enterprise IT Tool. — The IT Portfolio Management Automation project awarded 
a contract to develop the FBI’s Enterprise IT tool. This is a software package 
that will identify and track IT projects with baselined plans, schedules, and 
costs. It will also plan and track all FBI IT hardware and software infrastruc- 
ture procurements at an integrated, enterprise level. 

— Capital Planning and Investment Management ! Project Assurance. — The Invest- 
ment Management/Project Review Board now reviews and approves new IT in- 
vestments at specified stages of each IT project’s life cycle. We are also in the 
process of evaluating the FBI’s 130 + existing IT projects for overall health and 
placement within the system development life cycle. This will enable FBI execu- 
tives to uncover and address cost, schedule and performance risks. IT Invest- 
ment Management will use our Enterprise IT Tool to track new FBI IT invest- 
ments to ensure alignment with mission goals. 

— Performance and Results-Based Management (IT Metrics). — We are updating an 
IT Metrics program that identifies and measures IT performance according to 
industry standards, government regulations, and Earned Value Management 
System (EVMS) principles. Currently, we publish a CIO Monthly IT metrics re- 
port using the Balanced Scorecard Methodology. Our plan is to establish EVMS 
for “major” IT projects. When a program or project metric varies by more than 
10 percent of the acceptable thresholds for cost, schedule, and performance, it 
will trigger closer scrutiny and remedial action by the Investment Management/ 
Project Review Board. 

— Acquisition and Financial Reform. — IT Acquisition Reform, a joint initiative be- 
tween the CIO and the Chief Financial Officer of the FBI and DOJ, will stand- 
ardize and automate all procurement actions, involving all IT acquisitions, as 
well as focus on increased competition and small business involvement. In 2004, 
the FBI entered into multi-year enterprise-wide agreements with Microsoft, Or- 
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acle and Dell which have saved millions of dollars in licensing fees. The savings 
derived from these contracts have been reinvested into technology projects, such 
as SIPRNET and FAMS (FBI Automated Messaging System). SIPRNET gives 
the FBI desktop connectivity to the Intelligence Community and FAMS is based 
on the Defense Messaging System (DMS). The FBI is the first civilian agency 
to operate a classified DMS-like system. 

— Leadership. — We have begun to train our Program and Project Managers as 
well as executive management personnel to become certified as Program Man- 
agement Professionals (PMP), which is in compliance with the federal guidance. 
We currently have two certified Government and five contractor PMPs. Approxi- 
mately 25 managers have taken the PMP review course and plan to take the 
test. Another 20 are currently enrolling in the training program. This and other 
leadership training provides best practices and techniques to provide better 
management of the IT projects and the enterprise IT portfolio. 

— IT Policy. — We are in the process of updating a Master IT Policy List. Once es- 
tablished, any new IT policies or modifications will have to be reviewed and ap- 
proved by the IT Policy Review Board. The Master List will enable the CIO to 
monitor all IT projects during the Life Cycle Management Directive control gate 
review processes and enforce all applicable IT policies. 

— Technology Assessment. — The FBI’s Chief Technology Officer is working closely 
with the Enterprise Architecture team of the FBI and DOJ to standardize en- 
terprise technology standards, technical reference models, technical architec- 
tures, and technical design reviews under the Life Cycle Management Directive 
and system testing/integration. A unified test and integration facility will allow 
for centralized technology assessment that provides responsive IT solutions to 
meet mission goals. These measures mitigate project risks through common, 
interoperable, supportable and affordable solutions. 

— Seeurity and Information Assurance. — We have implemented an Information 
Assurance Program which implements key IT capabilities such as Public Key 
Infrastructure (PKI) and the Enterprise Security Operations Center (ESOC), to 
strengthen IT services in the FBI and DOJ and mitigate internal and external 
threats. Certification and Accreditation is being required for all IT Projects and 
Systems to further mitigate project risk. 

CONCLUSION 

Mr. Chairman, in the aftermath of VCF, the FBI is faced with difficult decisions 
on how best to proceed with our evolving IT needs and evolving technologies. This 
Committee and the American people have my personal assurance that we will pro- 
ceed as expeditiously and as prudently as possible to provide our employees with 
the automated capabilities they need. We have expanded the team of IT profes- 
sionals within the FBI, each of whom has demonstrated an ability to perform under 
adverse circumstances. We have learned many valuable lessons over the past few 
years and, as a result, will be able to apply these lessons and avoid many of the 
pitfalls that befell this project in the past. 

I would like to close by thanking the Committee, and you in particular, Mr. Chair- 
man, for your support throughout this endeavor, and I look forward to working with 
you and your staff as we chart our course for the future. 

Senator Gregg. The FBI has obviously got a problem and you 
are willing to address it and you have been forthright in explaining 
it, but I do think it is important to go back to some of the causes 
of the problem and make sure that those things are being ad- 
dressed. 

The Aerospace review, and I think choosing Aerospace, from 
what I can figure out, was a reasonable choice. They are inde- 
pendent and they appear to be quite objective. But they have three 
basic findings. One, that the architecture was developed without 
adequate assessment of alternatives and conformance to various 
architectural standards and in a way that precluded the incorpora- 
tion of significant commercial off-the-shelf software. I want to get 
back to that point because I want to know if that was an inten- 
tional decision because it appears to have driven cost. 

Second, the high-level documents, including the concept of oper- 
ations, systems architecture, and systems requirement, were nei- 
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ther complete nor consistent and did not map to users’ needs, 
which I find unusual. 

And three, the requirements and design documentations were in- 
complete, imprecise, requirements in design tracing have gaps, and 
the software cannot be maintained without difficulty and is, there- 
fore, unfit for reuse. We are looking at the 2003 delivery, of course, 
but this was the format on which 2004 was, I presume, built out 
of And even if it wasn’t, it still raises huge issues since we paid 
$170 million to get it. 

And then Aerospace concluded that it would be better not to even 
develop it this way, that we should go to the off-the-shelf approach, 
which raises three fundamental issues which I am wondering how 
the FBI plans to approach them as it moves forward. 

The first one is, why didn’t we have in the FBI the technical peo- 
ple who would have picked up on things like failure of architectural 
design, failure to meet standards which were fairly consistent 
across the development of software architecture which weren’t 
being met? There was a huge turnover of people during this period. 
Is it possible for an agency like the FBI to maintain the quality of 
people that are necessary in order to monitor a program of this size 
or should they — do we almost as a matter of systems have to put 
that monitoring into an independent group in order to make sure 
that we have the talent necessary to double-check a contractor like 
this? 

Second, why would we ever choose a cost-plus contract? I mean, 
this experience of cost-plus is pretty horrific across Federal funding 
activities. 

And third, this point which Aerospace makes about actually de- 
veloping a software which wouldn’t conform or wouldn’t be inte- 
grated with off-the-shelf activity. We know by definition that tech- 
nology mutates constantly and improves. I mean, isn’t it inherent 
to any technological system of this size that you are going to want 
to be able to migrate to the next system, which is going to work 
better, and that next system isn’t necessarily going to be internally 
developed, it is going to probably be developed by some smart 
bunch of folks who spun off from Massachusetts Institute of Tech- 
nology (MIT) and are sitting in a garage somewhere in hopefully 
New Hampshire? 

But it is not going to probably come from within the agency be- 
cause you don’t have the time and you don’t have the people and 
you don’t have the talent. Or you have the talent, maybe, but you 
don’t have the time to focus on the mutation. 

So have we addressed those three issues which I see as systemic 
to the question of why it has failed? 

QUALITY OF PERSONNEL 

Mr. Mueller. Let me take a crack at them and then turn it over 
to Mr. Azmi. 

In terms of the quality of personnel we had in the Bureau, I had 
a CIO, a very excellent CIO for the first year after I was there. He 
then retired. I then went on a nationwide search for a CIO which 
took about 8 to 12 months. The persons who were proffered for a 
variety of reasons fell through and there was a gap during that pe- 
riod of time in leadership at the CIO position. That hurt us. 
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I also, perhaps due to my naivete, did believe that we had the 
appropriate program managers. I had persons in from other organi- 
zations such as IBM and Lucent. I came to find out that there are 
project managers in a particular skill that we needed. I did not pro- 
vide to our project managers or to the users group. It was a soft- 
ware engineer specialist with the capability of drilling down into 
that which was being composed by SAIC. 

Now, do I have that capability now? I don’t think, and I will ask 
Zal, I am not certain that we have that full capability to drill down 
into a particular software package and determine whether every- 
thing is going as it should go. 

I do know that we have greatly expanded our CIO office under 
Zal Azmi. One of the things that he has brought is the ability to 
give me the bad news early on. One of the problems of anybody 
who runs an organization like mine is that people want to give you 
the good news. They do not want to give you the bad news. He has 
always been out there giving me the bad news and he has brought 
on board a technology officer who is the type of person that goes 
out and looks at each one of these COTS products. 

All that being said, we will have to augment our staff with con- 
tractors. We will have to go and look, as we have in the past, for 
expertise outside the Bureau to make certain that we have covered 
all of these areas of expertise. 

COST-PLUS CONTRACT AND COTS PRODUCTS 

As to your second question, on a cost-plus contract, that was en- 
tered into in the summer of 2000. I do not have the facts or the 
understanding as to why we entered into a cost-plus contract in the 
summer of 2000, in the summer of 2001. I can tell you that my ex- 
perience is we will never again in the Bureau enter into a cost-plus 
product that can lead us so far astray. 

I will tell you that prior to the last piece of the second part of 
Trilogy, which was putting in the networks, the local area net- 
works, the wide area networks, at the secret level, at the classified 
level, which was a challenge, we had difficulties with the cost-plus 
contract with that contractor and ended up restructuring it so we 
got a commitment to produce at a particular cost at a particular 
time. 

Last, with regard to COTS products, as I become more knowl- 
edgeable about technology, it goes without saying, I think, that the 
world has come to be a plug-and-play world. You don’t get a full 
system of stereo television all in one package by one manufacturer 
now. What you have is plug and play, whether it be computers or 
your stereo or what have you. As we have grown since 2001, it is 
clear that in developing a package such as the Virtual Case File, 
we have to look at COTS products. We have to use COTS products. 
We have to phase it in, understanding that down the road 1, 2, or 
3 years hence, we may have to unplug a product and plug in a new 
one. 

Zal, do you have anything to add? 

Mr. Azmi. I want to add to the concept of cost-plus contract. The 
Bureau originally actually got into this contract in 2001 because we 
did not have all of our requirements defined. However, in 2002, 
there was a joint application development session between the Bu- 
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reau and SAIC and, at that point, we developed a solid base for re- 
quirements, and, at that point, that contract could have gone to a 
performance-based contracting. However, that contract continued 
as a cost-plus contract. 

I will say that in June 2004, when we decided to actually develop 
the initial operating capability, we did move to a performance- 
based contract. That is the main reason why the software was de- 
veloped on time and within the budget. 

I would also add that even though IOC is only 10 percent of the 
VCF, I think the concept is sound and we can implement that for 
larger contracts. 


ENTERPRISE ARCHITECTURE 

The other question, Mr. Chairman, you had was about enterprise 
architecture and what we are doing, where we are going from here. 
I submit to you that we have already solidified our requirements 
for Virtual Case File or a case management system of the future. 
We have already mapped those requirements through a Federal en- 
terprise architecture framework, which is the best practice, is the 
standard the Federal Government uses. We have already mapped 
our software, or our requirements to what they call a service ref- 
erence model. We have already done this mapping. 

That will enable us to actually deliver a case management sys- 
tem of the future in phases, with capabilities being available to the 
users shortly after the contract is awarded, and that is the concept 
we are going to move forward with, the small deliverables and the 
contained time with program management and project manage- 
ment disciplines in place. 

HOW DO WE GET THE MONEY BACK 

Senator Gregg. I want to make sure everybody has time here so 
I will reserve my questions, but I am sure somebody is going to ask 
you how we are going to get any of this money back and that is 
a question I do hope we get to. 

delivery elements of VCF ON TRACK 

Senator Leahy. Thank you, Mr. Chairman, and I would ask also 
consent that I have a little time to put my full statement in the 
record and keep this short. 

Senator Gregg. Of course. 

Senator Leahy. The reason I ask is that it would be somewhat 
more lengthy because it also involves my other hat on authoriza- 
tion. 

Director Mueller, on May 20, 2004, you testified before the Judi- 
ciary Committee. You stated, “We are on track to deliver elements 
of Virtual Case File capabilities by the end of this year.” I re- 
sponded to that and I said, “What elements and what do you mean 
by elements?” I don’t think I ever got a clear answer on elements, 
but you did say, quote, “We are in negotiations with our contractor 
on finishing out that last part of the Trilogy project, the Virtual 
Case File, and my hope and expectation is that that will be com- 
pleted by the end of this year. But I do believe that when we are 
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concluded this year” — 2004 — “we have the foundation for cutting- 
edge technology for an organization our size,” close quote. 

At the same hearing in May 2004, Senator DeWine of Ohio asked 
you this. Quote, “Do you currently have enough money to complete 
Trilogy? What will be the total cost of Trilogy? How much money 
do you have left to spend on the program, and when will Trilogy 
be completed?” You responded, “I believe we do have sufficient 
money. I believe the total cost will be close to $560 million. And 
the last piece of Trilogy, that is the Virtual Case File, my expecta- 
tion, it will be in by the end of this year.” Senator DeWine said, 
“End of this year?” You responded, “This year.” 

Now, we do know that by the time you testified in May 2004, al- 
most 1 year ago. Virtual Case File was already on life support. The 
FBI had already twice rejected SAIC’s delivery of the Virtual Case 
File. It already identified nearly 400 potential problems with the 
software. It had already been told by Virtual Case File that cor- 
recting these problems would cost an additional $56 million and an 
additional year. As you say in your testimony today, they are both 
unacceptable to the FBI. 

In addition, the FBI was already negotiating for a scaled-down 
version of VCF, the initial operating capability of VCF Light. 

Just the day before the hearing when we asked you these ques- 
tions where we got a pretty rosy scenario, the FBI submitted a re- 
quest, Federal Systems Integration and Management (FEDSIM), 
the contract manager, to estimate the cost associated with shutting 
down 90 percent of it. 

Now, I don’t know anybody who has been more supportive in the 
30 years I have been here of the FBI than I have. Others have been 
as supportive. I don’t know of anybody more supportive. I have 
been extremely supportive of you. But I am ready to tear out what 
little bit of hair I have left. 

Why didn’t you mention any of these problems, all of which were 
there, when you were asked about the status of the project in May 
2004? You had a friendly audience. You had me. You had one of 
the leading Republicans, Mike DeWine. We were asking you these 
questions, and the answers we got didn’t comport with the facts. 
Why? 

DIRECTOR MUELLER’S RESPONSES 

Mr. Mueller. Senator, I don’t want you to lose the last of that 
hair. 

Senator Leahy. There is not much left, I can tell you right now, 
nor is there any more patience. 

Mr. Mueller. I will tell you, as we went through the spring — 
and I would have to look at the dates — as we went through the 
spring last year, I had voices telling me, particularly from SAIC, 
that they could produce. I met with the Chief Executive Officer 
(CEO) in the spring — I am not certain of the date — and received 
from the CEO the assurances that we could — and by “we,” I mean 
SAIC would produce and it was my expectation that we would have 
a substantial portion, not all, but a substantial portion of Virtual 
Case File by the end of the year. 

Now, when that came in terms of the timing of my testimony, I 
am not certain. On the other hand, I will tell you that Zal Azmi 
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has always raised questions about this. I knew that there were 
issues with regard to the project as it was given to us in December 
2003, but I had already been through a similar circumstance with 
Computer Sciences Corporation (CSC) in which we had to renego- 
tiate, we had to go back to the drawing table, and they came 
through under budget, on time, as we had done so. And there was 
a part of me in the spring of 2004 that thought that we could go 
through exactly the same exercise. 

FEDERAL SYSTEMS INTEGRATION AND MANAGEMENT REQUEST 

Senator Leahy. The day before the hearing, the FBI had sub- 
mitted a request to FEDSIM asking, what would it cost to shut 
down 90 percent of it. 

Mr. Mueller. I am not familiar with that. I am not certain I 
was familiar with that at the time. 

Senator Leahy. I hope not, because if you were familiar with it, 
your answers to mine and Senator DeWine’s questions were totally 
inconsistent with what the facts were. 

And then we sent follow-up questions to you. I did and several 
others did. You told me you completed your responses some time 
ago and sent them on to the Department of Justice for review. It 
has been 8 months. I don’t know who is good cop/bad cop, to use 
an analogy in your business, who is good cop/bad cop here, but we 
asked specific questions. The answers we were given did not com- 
port with the facts, and I will accept your statement here today 
that you were not aware that the day before, they were trying to 
figure out how to close down 90 percent of it. 

But the answers — somebody has got to bear responsibility. It 
can’t just simply be, well, the Department of Justice told us for 8 
months, don’t answer these questions. We are talking about hun- 
dreds of millions of dollars and a friendly committee. What in the 
hell goes on if it is an unfriendly committee? 

Mr. Mueller. Is that a question. Senator? 

Senator Leahy. Yes. When are we going to get the answers? 

Mr. Mueller. Well, as I indicated to you yesterday, the answers 
were provided to the Department of Justice in October. We have 
been working with them. I am as frustrated that you do not have 
the answers as you quite obviously are and I am certainly willing 
to do what I can to work to get those answers to you. 

BUDGET TO COMPLETE TRILOGY 

Senator Leahy. Well, let me ask you a specific question for ap- 
propriations. Does the FBI have sufficient money to complete Tril- 
ogy, including VCF or a similar case management system, or will 
the FBI reprogram or request additional funds to fix and find a re- 
placement for Virtual Case File in this upcoming budget cycle? 

Mr. Mueller. What we are planning to do is utilize funds that 
we have outstanding for this fiscal year and in 6 to 8 — and correct 
me if I am wrong, Zal, on this — and in 6 to 8 weeks, we ought to 
have a better feel for what it would cost to bring on the various 
components that we are anticipating bringing on in the phased-in 
development of Virtual Case File. It would not be a 1-year phase- 
in. It would be a 2- or a 3-year phase-in. At this point in time, hav- 
ing just received the Aerospace report, we are examining all of our 
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options and it will be at least 6 to 8 weeks before we can come back 
to you and lay out in front of you our strategy and say, this is what 
we want to do. These are the COTS products we may want to use 
and this is what it will cost. 

I am looking to reprogram funds to do it, certainly within this 
fiscal year, and then we will look at where we are when it comes 
to 2006-2007. 


REPROGRAMMING 

Senator Leahy. Well, if you reprogram the funds 

Senator Gregg. Senator, if I can just interrupt, I think it is im- 
portant to note this phased-in development issue, because this com- 
mittee was actually very aggressive with the FBI saying that this 
program should have been phased in at the beginning 

Senator Leahy. I remember that. 

Senator Gregg [continuing]. As I think the Director will recall, 
and so I think at least they should be credited with the next steps 
they are going to do phases. 

Senator Leahy. But then on that, where are you going to repro- 
gram the money? Does that mean you are going to reduce other 
programs? 

Mr. Mueller. We have carryover money of approximately $15 
million and we are looking at other savings that we have managed 
to put into Virtual Case File, or what will become Virtual Case 
File, and we are also going to look at reprogramming additional 
funds, depending on what we can do and how fast we can do it in 
this fiscal year. 

Senator Leahy. Will you report to this subcommittee — well, the 
reprogramming, you will anyway 

Mr. Mueller. Absolutely. 

Senator Leahy [continuing]. But will you report to this sub- 
committee from what programs you are finding savings? 

Mr. Mueller. Yes. 

Senator Leahy. You understand the danger of that, of course. 

Mr. Mueller. Yes. 

Senator Leahy. All right. 

Mr. Mueller. I would anticipate we would have to. We repro- 
gram — if it is over a certain amount, we are up here in any event, 
so 

Senator Leahy. We are just curious 

Mr. Mueller [continuing]. It is an ongoing 

RECOUPING FUNDS FROM SAIC 

Senator Leahy. We are just curious what programs that we have 
already authorized might get cut back or eliminated by a re- 
programming to take care of the mistakes in the VCF. By the way, 
speaking of money, do you have plans to recoup funds from SAIC, 
and if so, how much? 

Mr. Mueller. We have referred the matter over to the Depart- 
ment of Justice to look at, explore our options. 

Senator Leahy. Are they going to get an answer back to you 
quicker than they do to those of us in the Congress? 

Mr. Mueller. All I can tell you is we referred it to the Depart- 
ment of Justice, Senator, looking at to what extent either of the 
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parties are culpable. I do believe there is culpability, as I indicated, 
on both sides. I am not going to stand here and say that we are 
not in some part responsible for the fact that it was not brought 
home on time. But as I say, I believe SAIC was also responsible. 
The report from Aerospace seems to indicate some of those defi- 
ciencies and we are looking at our options to recover some of that 
money for the taxpayer. 

Senator Leahy. Do you have any estimate of how much that 
might be? 

Mr. Mueller. I do not. 

CASE MANAGEMENT 

Senator Leahy. Okay. Let me ask you just two questions and I 
will submit the rest, which is always scary because I will probably 
never get the answer, but when will agents have a functioning case 
management system in their hands? 

Mr. Mueller. A basic case management system, and there are 
various aspects to it — monitoring evidence, leads management, and 
the like, but a basic case management system, certainly we hope 
within 1 year. And I will tell you, I am guilty of 

Senator Leahy. One year from today? 

Mr. Mueller. Yes. And I am guilty in the past of raised expecta- 
tions. I thought we were going to produce. Every time I have gone 
to an office to talk to our people, I will talk about the importance 
of technology, the desirability of bringing us into the digital age, 
and have given them the expectation that we would have had Vir- 
tual Case File certainly by now. I went out and retrained a number 
of agents in support of Virtual Case File. So I am very reluctant 
to give estimates, understanding that I have been proven wrong in 
the past and I have raised expectations, not only of the agents but 
also of Congress and others who are interested in moving us into 
the digital age. 

INFORMATION TECHNOLOGY DEVELOPMENT AND FUNDS RECOVERY 

Senator Leahy. I will ask just one last question and I will submit 
the rest, and I ask this question because the same frustration — the 
biggest frustration I have in being unable to get answers is that 
over and over again on the things that we legitimately ask ques- 
tions about, either the Appropriations Committee or the author- 
izing committee, we don’t find out until we read it in the paper. 
We either find out because a newspaper reporter is able to get 
more or a TV reporter, or somebody has leaked something to them. 

So let me ask you this. Are there other clouds on the horizon 
with respect to the information technology efforts that you might 
like to tell us about today before we read about it in the press in 
the future? 

Mr. Mueller. That is a very broad question. 

Senator Leahy. I know it. It is a very broad subject. 

Mr. Mueller. Are there any clouds on the horizon with regard 
to the development of these systems? With regard to the develop- 
ment of these systems, I think the last piece of Trilogy was Virtual 
Case File, and I think you know exactly what we know with the 
various reports. We, upon occasion, have other areas in which tech- 
nology is affected. We are currently looking at an issue that does 
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not relate at all to our sensitive material — well, our classified ma- 
terials, but is an issue which I probably should raise to you in pri- 
vate. 

Senator Leahy. Okay. Fair enough. Will you? 

Mr. Mueller. I will. 

Senator Leahy. Thank you. Thank you, Mr. Chairman. 

Senator Gregg. Senator Mikulski. 

Senator Mikulski. Thank you very much, Mr. Chairman. 

Well, under this rock is another rock and under this rock is a 
black hole. The future — I am concerned. First of all, we can look 
back, but my concern is how do we move ahead. 

Can you tell me, number one, are you thinking about scrapping 
the program now that we have invested $170 million into it, or how 
much of the $170 million are we able to kind of recapture and get 
value for the agents to have what they need in the field? Are we 
just bagging it? We have got so many contractors there. You have 
got SAIC, and others working on the other parts of Trilogy and of 
course now you have Aerospace, the corporation’s comments and 
evaluations. Where are we here? What are you going to do? Are we 
scrapping a $170 million program here? 

Mr. Mueller. Let me start. Senator, by saying that the total 
contract was for $170 million. We think we can recover approxi- 
mately $53 million of that in terms of software, hardware that we 
have received in the course of that contract, so that will not be lost. 
We have in excess of $12 million left in the contract, which leaves 
approximately $104 to $105 million that we will not be able to re- 
cover. 


POSSIBILITY OF SCRAPING SAIC 

Senator Mikulski. Are you saying goodbye to SAIC now or are 
they going to be the ones that everybody walks into the woodshed, 
but then what happens after you come back from the woodshed to 
the main building? Are we going to get the case file 

Mr. Mueller. We are looking at all of our options and who 
can 

Senator Mikulski. So you don’t know who 

Mr. Mueller. We do not know who the contractor will be for the 
next phases of the program. Now, are we scrapping the program 
altogether, I think was one of your questions. 

Senator Mikulski. Yes. 

Mr. Mueller. The recommendation from Aerospace, based on 
their review of that which was provided to us by SAIC in December 
2003, was to scrap the project totally. We are looking at that. We 
are reviewing that. SAIC, I think, will tell you when they testify 
that the product that they have produced for us that is being tested 
down in New Orleans is state of the art. It is very good and we 
should adopt that. We are looking at that. 

On the one hand, SAIC says we have produced and the product 
we have got down in New Orleans is good and you ought to adopt 
that. On the other hand, we have the report from Aerospace that 
says, for a variety of reasons, you ought to scrap Virtual Case File. 
So we are evaluating those two 

Senator Mikulski. But SAIC says, we have delivered you an ini- 
tial product. It is now in New Orleans being tested. 
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Mr. Mueller. Yes, and it is good, state of the art 

Senator Mikulski. Well, wait. Wait. We don’t know yet. It is 
being tested. 

Mr. Mueller. That is what SAIC is saying. 

Senator Mikulski. It is being tested. 

Mr. Mueller. It is being tested. 

Senator Mikulski. So, number one, you don’t know whether you 
are going to scrap it or not, and if you do, whether you scrap it or 
not, moving ahead, you don’t know who the contractor will be. And 
if you don’t know who the contractor will be, then you don’t know 
how much it will cost 

Mr. Mueller. Correct. 


DECISIONMAKING 

Senator Mikulski. So this is not a happy situation. 

Mr. Mueller. No. I would agree with that. It is not a happy sit- 
uation when we are 

Senator Mikulski. And then my question becomes, then, who is 
in charge to get this back on track and what are your time tables? 
The chairman will have an appropriations deadline. We have a 
very tight budget — we have been faced with spartan allocations. 
And then who is going to be in charge to make all these decisions? 
And I know you are going to say you are in charge, okay. That is 
great. But like the Pope is in charge of the Catholic Church, who 
is in charge of this confessional? 

Mr. Mueller. Well, the way you put it, maybe I am in charge 
of the confessional, but I will rely on Zal Azmi and his team for 
advice and management of the process as we go forward. But as 
I said before and I have said since I have arrived, and I have said 
it in this context and other contexts, we need and would look to 
outside, independent advice on whether we are on the right track. 
We have had — and I have gone to any number of outside entities 
to get advice on whether we are on the right track, experts outside, 
and we will continue to do that. 

Senator Gregg. Senator 

Senator Mikulski. Well, my time is up 

Senator Gregg. No, your time is not up, but I am just wondering 
if I could interject a question here. 

Senator Mikulski. Please, yes. I think this will work best this 
way. 


EVALUATING THE 2004 PRODUCT 

Senator Gregg. Are you evaluating the 2004 product as it is now 
being used in a demonstration in New Orleans independently, and 
if you are, who is doing that? 

Mr. Azmi. That product in New Orleans is a prototype or a func- 
tional prototype of the VCF IOC, initial operating capability. That 
software is one-third of the — I am sorry, one-tenth of the VCF soft- 
ware. It is not all of the capabilities that was promised. It is just 
one-tenth of that. Within that software, the FBI has also included 
a number of capabilities that were developed by FBI staff, pro- 
grammers. So, that is a combination of two programs that is being 
tested in New Orleans. 
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By the end of March, we will shut down that evaluation period 
and will have 30 days to actually gather information and feedback 
from our users in New Orleans to see how they liked it. That is 
the work we are doing with our staff over in New Orleans, sir. 

Senator Gregg. Can I postpone you for one more question? 

Senator Mikulski. Sure. 

Senator Gregg. You are saying it is one-tenth of what was sup- 
posed to be delivered. 

Mr. Azmi. That is correct, sir. 

Senator Gregg. The project that was evaluated and found so 
lacking by Aerospace, which was the 2003 product, was that the 
entire product? 

Mr. Azmi. That is correct, sir. 

Senator Gregg. Thank you. 

Mr. Mueller. I think 

Senator Mikulski. Did you want to pick up on my question? 

Mr. Mueller. I think Mr. Azmi wanted to add on the answer to 
your question, if he could. 

Senator Mikulski. Yes. 

Mr. Azmi. I know that Director Mueller is taking responsibility 
for the program as a whole, but as the Chief Information Officer 
for the FBI, it is my responsibility to develop information tech- 
nology to our users. What steps have I taken since my arrival to 
actually make sure 

Senator Mikulski. When did you arrive? 

Mr. Azmi. November 2003, ma’am. 

Senator Mikulski. Thank you. 

Mr. Azmi. We have taken a number of steps to actually correct 
the deficiencies overall with information technology programs with- 
in the FBI. But specifically for the VCF program, what we have 
done, we have completed our requirements. We have a require- 
ments document for a case management system that our users, our 
agents, and our analysts want and the FBI. We have mapped those 
requirements toward services that are guidelines by the Federal 
enterprise architecture framework. We have those services. We 
have broken down those services into phases to ensure that we 
have the ability and capability to deliver those into phases. 

We have also asked another independent contractor to develop 
what we call an independent Government cost estimate to tell us 
exactly how much every one of these phases will cost. That report 
is due to the FBI by mid-February, ma’am. 

Senator Mikulski. I appreciate that answer. I know my time is 
about up 

Senator Gregg. You have as much time as you want. 

Senator Mikulski. Director Mueller, did you 

Mr. Mueller. I wanted to add one other thing that has become 
important. It was in the National Sciences report, and that is the 
necessity for an enterprise architecture for the FBI as a whole. We 
have never had an enterprise architecture. We have been 
stovepiped. And one of the things we have done over the last year 
is begin to develop an enterprise architecture so that whenever we 
bring on an information technology product, it fits within that en- 
terprise architecture. 
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For us to move forward, we have to have the enterprise architec- 
ture to assure that whatever we bring in is consistent with and 
works with other software and hardware packages that we may 
bring on board, and that is a substantial advance for us. We have 
a team working on it and I think we are on the track to have one 
of the better enterprise architectures for any institution in Wash- 
ington. 


ACCELERATED FUNDING AND OVERSIGHT 

Senator Mikulski. Well, I appreciate these answers and I cer- 
tainly your attempt, Mr. Azmi, to try to bring order out of chaos. 
I also appreciate the fact that after September 11, there was this 
incredible need to retool the FBI. There was an accelerated ops 
tempo, if you will, because we didn’t know when they were going 
to try to kill us again. We were still standing sentry because they 
might be trying to kill us again in an hour and a half. 

So we understand the challenges you faced, the FBI faced, and 
with this increased ops tempo, though, your Congress gave you 
money as well as in a variety of homeland security agencies money 
to protect the United States of America. That is what these files 
and all this technology is all about, is to maximize and leverage an 
agent to make that agent the most effective person that they can 
to do the mission. 

I am really concerned that after 3 V 2 years, where in the hell are 
we and have we just wasted money, have we just wasted time, and 
how we won’t repeat it again, because in the report, it talked about 
how the FBI had changing requirements. It is what we hear at the 
Pentagon. Every time they build a ship, they meet with an admiral 
and a boatswain’s mate and the requirements get changed. 

So my question — well, first of all, just know, I know you are dis- 
appointed and I am disappointed. I believe that this is a systemic 
issue with some of the accelerated funding in homeland security 
and I think calls for additional oversight in appropriations. 

But now having then come back to where we are, with the re- 
forms Mr. Azmi has put in to bring order out of chaos, when do 
you think you can tell the subcommittee what it is that you want 
to do and how much it will cost? 

Mr. Mueller. Two months. 

Senator Mikulski. Two months. 

Mr. Mueller. I think we will have a much better handle on 
where we are at that time. 

Senator Mikulski. Fine. But I think we also have to understand 
the pressure that you — when I say you personally, because we were 
together in some tough environments and I respect you very much 
and all the agents. But, wow, I think we kind of have to regroup, 
don’t you agree, Mr. Chairman? 

Senator Gregg. As usual, the Senator from Maryland has gotten 
to the essence of the issue. 

Senator Mikulski. Thank you. We look forward to working with 
you, Mr. Chairman, and we look forward to making sure there is 
not an empty chair here. 
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INDEPENDENT EVALUATING ASSISTIVE TEAM 

Senator Gregg. It would be very enjoyable were you in that 
chair. 

And just to follow up on the Senator from Maryland’s points, 
which I think are absolutely correct, and Senator Stevens actually 
made this point before he had to leave, this could be a systemic 
issue across other agencies, as we tooled up so quickly with tech- 
nology that agencies that didn’t have the personnel capability to 
properly manage this tooling up either bring online technology that 
can’t migrate into the greater needs, can’t keep up with the chang- 
ing times, or simply can’t do the job. 

That is why I get back to this issue of should we have an inde- 
pendent evaluating assistive team, where we have the level of ex- 
pertise there that is consistent and technically current to come in 
and help an agency like the FBI. I mean, you have got a good per- 
son in Mr. Azmi. I am extremely impressed with Mr. Azmi. I have 
had a fair number of discussions with him. But is the FBI ever ca- 
pable of getting out of the trees and looking at the forest on the 
issue of technology the way an independent group might be able to 
help you? 

Mr. Mueller. I think it is worth exploring. I think, as I have 
come to learn, that development of software for a particular organi- 
zation requires a complement of individuals within the organiza- 
tion who understand the work of that organization 

Senator Gregg. That is obvious. 

Mr. Mueller [continuing]. Usually called user groups, and the 
experts on the other side who know the technology. And the coming 
together of those two is exceptionally difficult. A third party with 
the expertise, or a third entity that could provide the expertise to 
an agency may be worthwhile. 

Right now, we understand we don’t have all the areas of exper- 
tise in the Bureau and we go out to outside contractors to bring 
that expertise in, in particular areas. But it is certainly something 
that perhaps should be explored. 

I will tell you also, in response to Senator Mikulski’s point about 
pushing hard on the technology, one of the things that we did do 
which I think backfired on us is push hard after September 11 to 
get the technology on as fast as possible without understanding, 
fully understanding the detrimental side effects to pushing too 
hard to get that technology on board without going through, unfor- 
tunately, some tedious, time consuming steps in order to get what 
you need, even though you have to delay, and that is a lesson I 
have learned in the course of working with Virtual Case File. 

file management and wireless technology 

Senator Mikulski. Mr. Chairman, just to you, after 9/11 and 
then for those of us on the Intelligence Committee also authorizing 
and appropriating with the FBI, it was, in every one of the agen- 
cies where there was responsibility for protecting us against preda- 
tory attacks, there was this increased tempo and every desire to 
move quickly, even if we made mistakes. It was better to make a 
mistake and spend the money, but don’t dilly-dally on the process. 
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At the same time, we had that sniper in Maryland, and I wish 
you could have been there to see the FBI, Bureau of Alcohol, To- 
bacco and Firearms (BATF), hundreds of agents in a room about 
this size with wireless technology. That is when I got a sense of 
the files, the management, and the communication, how they all 
worked with all of the leads all over America, with BATF and the 
ballistic lab, and then local law enforcement. It was really stun- 
ning. And when we have the right tools, it is amazing. But again, 
they were at the edge of their chair, working with every tool at 
their disposal, and even though some of those tools were out of 
date. 

So again, we see the way they have to escalate to an intense 
level. They have an attitude which we appreciate. Damn the tor- 
pedoes. So if you make mistakes or you spend too much money or 
whatever, at least grab the sniper, grab the killer, grab the ter- 
rorist, grab the predator, and we have made mistakes. These are 
big-bucket mistakes, but now it is to regroup. 

But I think it wasn’t because there wasn’t a desire to move 
quickly and do a good job. I am not white-washing this, but 

Mr. Mueller. If I could respond briefly, Mr. Chairman, I think 
if you look at it as a continuum, after September 11, if you went 
into that room, you saw paper all around 

Senator Mikulski. You did. 

Mr. Mueller [continuing]. Because we would have to take down 
everything on paper and run it by paper. And if you went into our 
SIOC in the wake of September 11, you would find piles of paper 
around. 

We evolved. When we worked with the other agencies. Federal 
and local, it was pretty much a paperless organization, and we 
have evolved to be paperless when we have challenges such as 
that. 

Unfortunately, we had to run files between offices. We did not 
have the communications capabilities at the time of the sniper at- 
tacks that we would want, even though we had the paperless entry 
of information, and we have evolved yet from there. 

So we have made headway in a number of these areas that en- 
ables us, particularly with substantial challenges such as Sep- 
tember 11 or the sniper attacks and the like, to do our business 
digitally. 


CLOSING REMARKS 

Senator Gregg. Thank you Senator, which I think gets back to 
what our purpose here is, is to make the agent on the street more 
effective in protecting us. We know the commitment of the Bureau. 
We know it is extraordinary. We know the people that serve us 
there, including right up to yourself, are the best and trying hard- 
est and we respect that, but obviously the taxpayers want to make 
sure they get value for their dollar, as you do, too. So that is what 
this hearing is about. 

I thank you. Thank you for your time. I appreciate your courtesy 
in giving us so much of your time. 

Mr. Mueller. Thank you, Mr. Chairman. 

Senator Gregg. Thank you, Mr. Director, Mr. Azmi. 
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SUBMITTED STATEMENTS 

We have a bit of an issue here in that we have got a vote at 3:30 
and a 4 o’clock event that I have to be at because the leader told 
me I have to be there and I am a big fan of the leader. So I think 
I am going to have to recess this hearing and probably reschedule 
the second panel, which I regret, because I think SAIC has every 
right to make their case in the public. They have obviously got a 
case they want to make as to their views, and obviously we would 
like to hear from Aerospace and from the Inspector General. 

The statements from these organizations not appearing and a 
statement from Senator Grassley will be inserted in the record. 

[The statements follow:] 

Prepared Statement of Arnold L. Punaro, Executive Vice President, Science 
Applications International Corporation 

Chairman Gregg and Senator Leahy: It is a privilege to appear before you today 
to testify concerning our portion of the work on the Trilogy Project for the Federal 
Bureau of Investigation. Mr. Chairman, I ask consent that my entire statement be 
entered into the record and with your permission I am prepared to summarize. 

introduction and context 

At the outset, let us say clearly that SAIC understands and appreciates the over- 
whelming demands and difficulties that the FBI has faced since the attacks of Sep- 
tember, 11. While we disagree with the Bureau over aspects of the Trilogy program 
history, we have only the greatest respect for the dedication with which the Bureau 
has pursued its mission of defending our nation under the enormous, and sometimes 
conflicting, pressures that surfaced in the aftermath of the terrorist attacks. 

SAIC, with 45,000 employees, is the largest privately owned research and engi- 
neering firm and one of the largest government contactors in the nation. As em- 
ployee owners, we have prided ourselves since our founding 36 years ago on our 
ability to assist the U.S. Government on programs of national importance. Our dedi- 
cation to work that matters is further reflected in an aggressive and pervasive eth- 
ics program. How our company operates and how we are perceived are matters of 
vital, personal interest to each and every employee. We have grown to become a 
very successful and sought after company by providing quality products and cre- 
ating satisfied customers. 

In that respect, let me mention several major, illustrative software engineering 
projects successfully designed and deployed for the FBI to illustrate the work we’ve 
done. 

— The Combined DNA Index System (CODIS) is a national DNA database system 
for use by United States and international law enforcement authorities by cre- 
ating DNA profiles and by matching unknown profiles found in the course of 
criminal investigations to profiles stored in local, state, and national databases 
here and overseas. 

— The FBI Interstate Identification Index (Triple-I) is the U.S. national criminal 
history system that maintains more than 40 million data entries (the largest 
and most accurate criminal history database in the world) and is used every day 
by state, local and federal law enforcement agency in the United States. 

— The National Instant Criminal Background Check System (NICS) implements 
the Brady Act. SAIC was contracted to develop, deploy, maintain, and support 
the federal, state, and local governments in checking a citizen’s eligibility to 
purchase a firearm (handing in excess of 30 million purchases to date). It han- 
dled more than four million calls per year from firearms dealers checking pur- 
chasers against the national database. To quote Mr. Michael D. Kirkpatrick 
(FBI Assistant Director in Charge, Criminal Justice Information Services Divi- 
sion at the time of the work) in his letter of appreciation to SAIC in January 
2004, “Not only is the successful implementation of the NICS directly attributed 
to the hard work and dedication of the SAIC staff, numerous post-implementa- 
tion challenges were met head-on and overcome with SAIC’s support — you have 
been a trustworthy, customer-oriented partner.” 

— Law-Enforcement Online (LEO) is a 24 hours a day, 7 days a week, online, real- 
time, controlled-access web portal (more than 43,000 users) providing a focal 
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point for electronic communication, education, and information sharing for the 
law-enforcement, criminal-justice, and public-safety communities nationwide. 

In sum, SAIC comes to this issue with a record of outstanding achievement in 
challenging projects, including specifically for the Federal Bureau of Investigation. 
We point this out not to boast, but to provide the context for considering some of 
the issues that have marked the public discussion of Trilogy and the manner in 
which SAIC has performed on this contract. 

The Results and the Reasons 

— Trilogy began in a pre-9/11 world with very different circumstances and require- 
ments than those that exist now. 

— The events of 9/11 caused massive and continuing change in the project while 
the FBI dealt with enormous post-attack pressures and demands. 

— The FBI’s requirements for the project — the list of what the FBI wanted the 
project to have and do — grew and changed continually while turbulence in FBI 
program management worked against stability and definitive guidance. 

— A key FBI decision to drop a controversial, high-risk plan for a one-step conver- 
sion to a new system opened the way for a sensible developmental approach of 
incremental improvements in capability. 

— The FBI and SAIC renegotiated the contract in summer 2004, coming to firm 
agreement on requirements for the incremental improvement through what is 
called the Virtual Case File (VCF) Initial Operating Capability (IOC). 

— SAIC acknowledges some areas where we made mistakes and particularly 
where we failed to adequately communicate our concerns to appropriate levels 
of management, to include the Director of the FBI. 

— SAIC delivered, and the FBI approved and accepted, VCF IOC within the allo- 
cated budget and ahead of schedule to industry-standard quality, offering FBI 
agents significant new tools in their counter-crime and counter-terror roles. 

Currently, the contract has a negotiated value of $130.3 million and a funded 
value of $123 million. To date, SAIC has been paid $115.2 million. We expect to be 
paid the funded value of $123 million at completion. In conjunction with this work 
effort, the company has invested $3.9 million of its own money to support the Tril- 
ogy program. 

Aerospace Corporation 

Before presenting SAIC’s testimony about the course of its work on Trilogy in de- 
tail, I want to speak briefly to the report by the Aerospace Corporation. While we 
have not been given a copy of this report, we were allowed to read a copy last week 
at the FBI. We appreciate that opportunity. Aerospace Corporation did not inform 
us, nor attempt to discuss in any way its findings — a lapse we find both inexplicable 
and contrary to the practices of inspectors general, the General Accounting Office, 
and other scientific groups, who find that comments from those reviewed contribute 
to a more balanced and useful report. 

The Aerospace Corporation produced a report on the wrong software while failing 
to concentrate on central issues that determine system performance. 

Had they asked us for comment, we could have told them they examined the 
wrong software. Mr. Chairman, I mean that in a literal sense. Aerospace Corpora- 
tion explicitly evaluated a snapshot in time of the software as if it were a finished 
product when in reality, as everyone should have known, it was still being devel- 
oped. The Aerospace Corporation says it found “evidence of incompleteness” and 
“failure to optimize.” This is hardly unexpected in a work in progress that was still 
months away from its delivery date. In academic terms, it was as if we had been 
assigned a paper due December but then graded it the previous summer. 

The product we presented to the FBI in December 2004 is not the product evalu- 
ated by Aerospace Corporation. VCF IOC was rigorously tested and accepted by the 
FBI after meeting 100 percent of its requirements. 

Because the software evaluated was different from the software delivered, SAIC 
believes that the Aerospace Corporation report is not an adequate basis for deciding 
on a future course of action concerning VCF. 

This is not to say we accept Aerospace Corporation’s judgments about the product 
that was evaluated. We emphatically do not. The Aerospace Corporation is a na- 
tional asset in its realm of expertise: aerospace. The Trilogy project is something 
else, altogether. We respectfully — but strongly — urge this subcommittee to consider 
that Aerospace Corporation did not bring a sufficient understanding of the unique- 
ness, complexity, and scope of the FBI undertaking to evaluate our software prod- 
uct. 

Central to the Aerospace report is criticism of requirements documentation. Time 
and again, in the Aerospace report we reviewed, we saw instances where criticisms 
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about requirements were based not on the substance of the requirements or whether 
or not the product satisfied the requirements, hut rather on ancillary data such as 
syntax in documentation. How well the product satisfied requirements was not a 
part of their evaluation. Based on examination of the documentation they concluded 
they were not assured the product would meet requirements and went no farther. 

In particular, SAIC categorically rejects the assertion that its work lacked engi- 
neering discipline, an assertion that appears without support in the document we 
read. This kind of assertion, without rigorous — or even specific — support should he 
unacceptable in an endeavor of this importance. For instance, Aerospace Corpora- 
tion did not look at the software development folders, which are key documents on 
how the code was designed and written. These comprise the “Bible” for software de- 
velopers. In a football analogy, it was as if Aerospace Corporation was asked to 
scout another team which had made available its playbook. They didn’t bother to 
read it. In fact, they scouted the wrong team. 

Even so, Mr. Chairman, we would welcome the opportunity, late though it may 
be, to discuss the findings with Aerospace Corporation. It could only benefit the FBI, 
which is our aim here. 


SAIC’S PARTICIPATION IN TRILOGY 

The FBI’s Trilogy program is a massive, multi-part, multi-contractor program for 
broad-based modernization and improvement of its information technology. In June 
2001, SAIC was competitively awarded a cost-plus-award fee developmental contract 
for the Trilogy User Application Component (UAC). This is an appropriate contract 
type because the project involved first working with the customer to develop and 
agree on what was needed (the requirements) and then execute the agreed tasks. 
The complexity and uniqueness of the missions of the Bureau also argued for this 
approach. Some of the public discussion of the Trilogy contract has been conducted 
as if the required tasks were well known at the start, and easily achievable. At no 
point in time has either condition existed. 

At the time of award in June 2001, the contract scope for SAIC called for develop- 
ment of a web front-end to the existing legacy applications used to manage case in- 
formation. When this effort was complete, SAlC was to define an Enterprise Case 
Management System. This was a measured low-risk approach building on existing, 
or legacy, systems within the Bureau. 

The attack of 9! 11 

The September 11, 2001, attacks had as profound an affect on this project as it 
did elsewhere in the nation. Following 9/11, the Bureau faced enormous and some- 
times conflicting pressures. Prior to the attack, the Bureau was dealing with revela- 
tions that a spy, Robert Hansen, had plundered FBI secrets. Security and integrity 
of information is a fundamental issue for the FBI. After the attack, it faced three 
often conflicting demands: 

— The need to share information in the post-9/11 world so authorized personnel 
could both see and connect the dots to analyze and exploit intelligence. 

— The need, in the post-Hansen world, to prevent all but a few specifically author- 
ized people from seeing truly sensitive information. 

— The need to ensure admissibility of investigative information in court in keeping 
with the complex body of legal, policy, and Attorney General Guidelines under 
which the Bureau operates. 

Thus, the FBI faces a task of great difficulty and complexity in building an infor- 
mation technology system that simultaneously meets all three imperatives. 

Trilogy after 9111 

Following the attack, the Bureau fundamentally reexamined the project. The ear- 
lier, measured approach of June 2001 called for improving legacy systems. In the 
wake of the attack, the FBI correctly determined that the legacy applications should 
be replaced to make the Bureau more effective in responding to terrorists’ threats 
as well as to improve the efficiency of the continuing criminal investigative mission. 

In the months following 9/11, the Bureau conducted an independent review of 
available Commercial Off-The-Shelf (COTS) systems and Government developed sys- 
tems, and determined they could not satisfy the requirements. Therefore, SAlC was 
tasked to in February 2002 to develop the replacement for the legacy systems using 
the original contract. The SAIC UAC contract was restructured to incorporate an 
aggressive development plan first conceived in February 2002. This became the elec- 
tronic Virtual Case File (VCF) contract. Thus, the FBI shelved 6 months of work 
that no longer fit the post-911 world, and directed SAIC take on a much more ambi- 
tious, high risk project. 
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The Trilogy VCF was a large and complex enterprise-level undertaking. There are 
no other criminal investigative management systems of this scale in the world. In 
terms of size, the VCF DELIVERY 1 system was to manage millions of case files 
on Day One with an annual growth of hundreds of thousands of cases per year. At 
start-up, the VCF DELIVERY 1 system was to store and index more than hundreds 
of millions of documents in a wide variety of formats. The VCF DELIVERY 1 system 
would support 30,000 users geographically dispersed across the United States and 
other countries. FBI agents, analysts, and support personnel would rely on the VCF 
DELIVERY 1 to conduct nearly all the business functions that support the criminal 
investigative process. The VCF DELIVERY 1 was also to provide hundreds of inter- 
faces to legacy systems. The VCF DELIVERY 1 system would manage this workload 
while providing a 3-second response to users as well as high system availability. 
This would not be an ordinary case file management system. 

The VCF was intended, in sum, to provide the next generation system supporting 
the FBI’s case file management concept. It would be, as the Justice Department In- 
spector General has reported, “the first real change in the FBI’s workflow and proc- 
esses since the 1950’s”. The VCF would move the FBI from its slow, paper-based 
processes into the twenty-first century with electronic work flow. VCF, it was envi- 
sioned, would support real-time coordination among agents, allow secure access to, 
and reporting of case information for all those authorized to receive it, regardless 
of organization or location. VCF would support a dispersed community of users in 
creating, accessing, and managing centrally stored electronic case file information. 
It would provide the foundation upon which the FBI could migrate its disconnected 
business processes into an integrated and seamless work environment. 

Following the 9/11 attacks, time was of the essence. SAIC was asked to devise 
an approach to deliver VCF in record time — on an even more aggressive schedule. 
The new challenge was to define, develop, and deploy a bureau-wide enterprise-level 
case management system in just 22 months. Without defined requirements or an en- 
terprise architecture for the FBI IT systems, this was a high risk approach that re- 
flected the post 9/11 atmosphere. Here is where SAIC made honest mistakes. We 
should have made known that this approach was too ambitious. 

VCF and “flash cutover” 

One of the key issues in the new VCF development strategy was the so-called 
“flash cutover” approach. That meant, simply, that the new VCF, in spite of its then 
undefined requirements, would not be implemented via a low risk, evolutionary 
strategy, but rather would be built as a grand design in record time and be imple- 
mented all at once in a “flash cutover” from the legacy systems to the new VCF. 
SAIC informed the Bureau this was a high-risk strategy. It was here that SAIC 
should have made its concerns known to the Director. The FBI insisted on this ag- 
gressive approach because of its critical need to improve information sharing and 
case management. SAIC agreed to undertake the challenge. In hindsight, this ap- 
proach was a fundamental error and, in May 2004, the National Research Council 
Computer and Telecommunications Science Board was highly critical of the flash 
cutover approach and instead argued in favor of an incremental deplo 3 mient model 
with prototyping and adequate time for test. From 2002 through mid-2004, the Bu- 
reau was committed to the flash cutover approach; however, after the Academy re- 
port, the Bureau agreed to a low-risk, incremental strategy. 

During 2003 and 2004, the Bureau’s understanding of how it should respond, of 
what mechanisms and process it might need, and how it should adjust the IT infra- 
structure to meet the challenges of fighting terrorism continued to evolve. Not sur- 
prisingly, the impact on the VCF program was continuing and significant. In the 
testimony of the Department of Justice Inspector General before this Subcommittee 
in March, 2004, the IG identified “poorly defined requirements that evolved as the 
project developed” as one of the reasons for the delays and cost increases in the Tril- 
ogy project. In fact, as recently as 4 months ago, the FBI had a team working to 
define, confirm, and refine their case management requirements. 

When the flash cutover approach was adopted, SAIC formulated an approach to 
meet the aggressive schedule. SAIC used eight development teams working in par- 
allel and a program staff that reached 250 full-time equivalents. The risks associ- 
ated with the multi-team, parallel approach became apparent in the fall of 2003. 
With multiple teams working on vertical slices of the system at breakneck speed, 
SAIC did not adequately enforce coding standards across the teams and this re- 
sulted in less than uniform code. In addition, this approach resulted in some level 
of duplication of effort in the code with different approaches used to solve similar 
problems. This, however, did not compromise the system. 

Another matter affecting the VCF software development was significant manage- 
ment turbulence. Since November 2001, there have been 19 Government manage- 
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ment personnel changes that had a direct and significant impact on the manage- 
ment of this project (11 FBI Changes and 8 FEDSIM Changes). This lack of con- 
tinuity among key Government managers contributed to the problems of ensuring 
the effective and timely implementation of this system. Each change brought new 
directions, a different perspective on priorities, and new interpretations of the re- 
quirements. 

In its report on Trilogy last year, the National Research Council spoke directly 
to the difficulty of developing software in the absence of specific, settled require- 
ments. As the Council noted, “[I]t is essentially impossible for even the most oper- 
ationally experienced IT applications developers to be able to anticipate in detail 
and in advance all of the requirements and specifications.” 

Probably the most damaging aspect of this development environment was the 
ever-shifting nature of the requirements. SAIC development teams would meet with 
the FBI agents assigned to the project to elicit system requirements, then SAIC 
would translate that into software designs. Often, however, the agents would look 
at the development product and reject it. They would then demand more changes 
to the design in a trial-and-error, “we-will-know-it-when-we-see-it” approach to de- 
velopment. The turbulence was not limited to the immediate changes demanded. 
They would ripple though the related parts of the software design. This cycle was 
repeated over and over again and prevented SAIC from defining system acceptance 
criteria and suitable test standards until requirements were finally agreed under 
VCF IOC this past summer. SAIC expressed concern over the affect of these 
changes on cost and schedule; however, we clearly failed to get the cumulative effect 
of these changes across to the FBI customer. We accept responsibility for this failure 
to elevate our concerns. 

The most significant of these changes, occurring during the period when the flash 
cutover strategy was in place, was to the Records Management System. SAIC had 
actually selected a commercial off the shelf (COTS) solution and the FBI had agreed 
to it. Then, late in 2003, FBI representatives decided they wanted a different ap- 
proach, which would require changes to another COTS software package. The new 
COTS vendor would not be able to modify the software until a new release of the 
software was available in spring 2004. At this point, the grand design approach of 
the flash cutover strategy had begun to fall apart. 

In December 2003, we delivered an evaluation copy of the VCF system. The FBI 
reviewed the product and identified 17 deficiencies, some of which were actually 
more changes in requirements. These deficiencies and changes were addressed by 
SAIC, and an updated version of the system was provided in March 2004. The FBI 
then asked SAIC to assess the cost and schedule impact of incorporating accumu- 
lated changes and finishing Delivery 1. SAIC complied with this request in April 
2004, but the FBI chose not to undertake this course of action. The goal established 
early in 2002 — define, develop, and deploy a bureau-wide, enterprise-level case man- 
agement system in 22 months — was now clearly in jeopardy and behind the aggres- 
sive schedule. 

From VCF to VCF IOC 

In May, 2004, a series of meetings between SAIC, the FBI, and FEDSIM took 
place to define a new strategy. What emerged from these meetings was a signifi- 
cantly different plan. 

In these meetings, the Bureau agreed to modify its flash cutover approach in 
favor of an incremental approach, allowing deployment of new capabilities. Second, 
instead of replacing its legacy systems at this juncture, the Bureau agreed to focus 
on creating new capabilities based on legacy systems. Finally, the new approach was 
christened VCF Initial Operating Capability (IOC) and it was set for Delivery in De- 
cember 2004. The fundamental understanding between the SAIC senior leadership 
and Director of the FBI that enabled SAIC to go forward on the VCF IOC was 
agreement, for the first time, on a fixed set of requirements and defined acceptance 
criteria. 


WHAT THE FBI RECEIVED IN VCF IOC 

In December of last year, SAIC delivered VCF IOC. The project was successful. 
It delivers significant new capabilities, complied with the December, 2004 delivery 
date, was within the budget allocated for IOC, met 100 percent of requirements es- 
tablished by the FBI for IOC, passed a rigorous testing phase, was accepted by the 
FBI, meets or exceeds industry standards for quality, and, most importantly, is 
working well today for FBI agents in New Orleans and Washington Headquarters. 
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Functional capabilities 

With VCF IOC the FBI has a system that will move agents from a slow, paper- 
based system to a twenty-first century system for their key investigative efforts. In 
the past investigative information was often held-up in Field Offices, captured in 
agent notebooks, stored away in filing cabinets, and generally held in different ways 
and different means all across the country. VCF IOC makes critical information 
available instantaneously, in a uniform, easy-to-access manner, to all who need to 
access it regardless of their physical location. Additionally, these new capabilities 
build a foundation for migrating now-disconnected business processes into an inte- 
grated work environment and provide the infrastructure required to add the addi- 
tional case management capabilities. Specifically, the functional capabilities of IOC 
include: 

— Investigative document import for the FD-302 and related documents (the cur- 
rent mainstay of FBI investigative effort) and National Security Letters. 

— Electronic workflow, validation, and approval meeting legal, policy, and Attor- 
ney General Guideline standards to ensure admissibility in court. 

— Upload of approved investigative documents into the appropriate case files as 
serials in the legacy Automated Case Support (ACS) system. 

Infrastructure capabilities 

If widely deployed, the infrastructure capabilities within IOC would take the Bu- 
reau from its current paper-based circumstances into a modern web-based environ- 
ment. Specifically, IOC delivers: 

— A modern 3-tier web based computing infrastructure (as a migration target from 
the legacy mainframe). 

— An effective web-based user interface, already well received by agents who have 
seen and used it. 

— Organizational Hierarchy maintenance infrastructure, which matches IT infra- 
structure to the Bureau’s organization. 

— Automated interface to the legacy ACS. 

— A significant part of the underlying infrastructure for security, access control, 
auditing and logging. 

— System management and integration with the FBI’s Enterprise Operations Cen- 
ter (EOC), a 24-7 monitoring and support center. 

The functional and infrastructure capabilities in IOC enable the rapid expansion 
of VCF capabilities, both to add new features and to integrate software developed 
for Delivery 1 but not included in IOC. As evidence of this, in November 2004, the 
FBI tasked SAIC to extend the capabilities of the IOC system to provide a signifi- 
cantly broader capability to the Agent users. These extensions were successfully im- 
plemented in less than three months and provided to the FBI pilot users, where 
they have been quite well received. 

We believe the FBI would be well served by expanding these capabilities beyond 
the pilot sites, even as an interim solution to its urgent needs. 

Beyond the capabilities and infrastructure active in IOC, SAIC has done substan- 
tial work toward meeting the full set of requirements articulated to date for the Bu- 
reau and enterprise-wide version of VCF. The product of that broader work can be 
categorized in three groups. In the first category are capabilities where implementa- 
tion was complete (or nearly complete), where integration and test were underway, 
and where routine software problems were being identified and fixed. These spe- 
cifics of work done in these categories include: 

— Case Management 

— Leads 

— Intake and Report of Investigative Activity (RIA) — which is a different way of 
approaching the import documents in IOC 

— Document Management 

— Notifications and Ticklers 

— Source Management 

— Text Search 

— Most of the Reporting Generation Capabilities 

— Case Classification Hierarchy Maintenance Infrastructure 

— The remainder of the underlying infrastructure for security, access control, au- 
diting and logging including complex business rules address the potentially con- 
flicting pressures to share information post-9/11 and to implement need to know 
restrictions post-Hansen. 

Beyond completing the integration and test effort, additional work would be re- 
quired to deploy these capabilities focused on (a) resolving outstanding requirements 
or implementation issues, and (b) adapting the capability away from the flash cut- 
over approach to the incremental deployment strategy. 
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The second category represents capabilities where implementation was in 
progress but engineering or requirements issues required resolution before imple- 
mentation could be completed, including: 

— Evidence Management 

— Analysis and Techniques and the remainder of the report generation capabili- 
ties. 

— Name search 

— Resource tracking and management 

— Crisis Case management 

The third category includes capabilities that were late requirements additions or 
implementation approach changes and preliminary engineering efforts were in 
place. This would include records management. 

In addition to these capabilities, SAIC performed substantial analysis and engi- 
neering efforts to document the complex and largely undocumented legacy environ- 
ment that has evolved over the years. That effort was critically important to the 
FBI’s information technology initiative. In a December, 2002 report, the DOJ IG 
noted that the lack of documentation for the legacy systems would limit “how rap- 
idly UAC can be developed and deployed” since “the FBI must know what it has 
before it can define the right solution to fix the problem”. The SAIC team made sig- 
nificant progress in this area producing 

— Over 300 Interface Control Documents (ICDs) covering the interfaces between 
internal FBI systems and also with external systems. 

— Extensive analysis and mapping of largely undocumented legacy data to a rela- 
tional model in preparation for migration into VCF. 

CONCLUSION 

In conclusion, SAIC has spent the last 36 years working hard and ethically to 
support important work for the U.S. government and our nation. We have been suc- 
cessful because we have delivered good work for our customers. We followed a dif- 
ficult path to get there. The Bureau faces difficult choices in difficult and chal- 
lenging times. Unfortunately, the flawed report from Aerospace Corporation does 
not provide a sound basis for making decisions about VCF IOC. 

The information technology assignment that the FBI envisioned and that SAIC 
accepted in June of 2001 changed dramatically after the terrible events of 9/11. As 
the FBI struggled to respond to new missions and conflicting demands, new tech- 
nology requirements also evolved, and we attempted to keep up. Finally, it became 
clear to all that the grand design envisioned in the full version of Virtual Case File 
was collapsing. The FBI agreed, instead, to an incremental approach that would — 
and did — produce immediate and tangible results. With the delivery of VCF IOC, 
SAIC has given FBI agents new capability today — not at some uncertain point years 
from now, but today as they work to combat both crime and terror across this na- 
tion. 

SAIC pledges to the Committee and to the FBI that we stand ready to work at 
cost with all parties to recognize the full potential of all of the extensive documenta- 
tion, analysis and code that has already been provided to further enhance the capa- 
bilities of the FBI to perform its vital tasks. 

If the FBI’s goal is to provide its agents enhanced capabilities as soon as possible 
and at relatively low additional cost, then we strongly recommend that the FBI con- 
tinue to deploy VCF capabilities to the agents using the highly successful incre- 
mental approach utilized for the VCF IOC delivery and to evolve it along with their 
emerging enterprise architecture. Using IOC should bring dramatic productivity im- 
provements now while the bureau develops a new system. 

If, however, the primary goal has shifted to meeting the new requirements of the 
new Federal Investigative Case Management System (FICMS), or to adopt the latest 
technology and COTS components that did not exist when VCF began, then the 
FBI’s agents will have to wait until these new programs deliver as yet undefined 
capabilities in three or more years. The Trilogy IOC provides much needed capabili- 
ties today that are scalable across the entire FBI and provides the foundation to 
quickly add other required capabilities incrementally over the next year. 


Prepared Statement of Gary P. Pulliam, Vice President, Civil and 
Commercial Operations, The Aerospace Corporation 

Mr. Chairman, distinguished committee members, and staff: I am pleased to rep- 
resent The Aerospace Corporation and appear before you today as you deliberate 
Trilogy and the Virtual Case File System. 
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As a private, nonprofit corporation, The Aerospace Corporation has provided engi- 
neering and scientific services to government organizations for over 40 years. We 
provide a stable, objective, expert source of analysis. We are focused on the govern- 
ment’s best interests, with no profit motive or predilection for any particular design 
or technical solution. 

As its primary activity. Aerospace operates a Federally Funded Research and De- 
velopment Center (FFRDC) sponsored by the Under Secretary of the Air Force, and 
managed by the Space and Missile Systems Center (SMC) in El Segundo, California. 
The Aerospace Corporation also undertakes projects for civil agencies that are in the 
national interest and are consistent with our corporate role. Over 350 staff members 
focus exclusively on computer systems, software, and information technology. 

Our unique “trusted agent” role provided to the Air Force has become known 
throughout the Intelligence Community. In executing our FFRDC mission, and more 
specifically, our support to the National Reconnaissance Office, our technical core 
competencies have become known to the FBI. 

1. INTRODUCTION 

In 2001, the Federal Bureau of Investigation (FBI) began a major information 
technology upgrade commonly known as The Trilogy Program. The User Applica- 
tions Component (UAC) is one of three basic elements of Trilogy. Organizations such 
as the Government Accountability Office, the Department of Justice Inspector Gen- 
eral, and the National Research Council have voiced serious concern about the 
progress in completing Trilogy, and specifically the UAC. In response to these con- 
cerns, the FBI developed and implemented a “corrective action plan” in June 2004. 
As part of the corrective action plan, the FBI requested that The Aerospace Cor- 
poration (hereafter. Aerospace), conduct an independent verification and validation 
of the UAC; specifically, the Virtual Case File (VCF) Delivery 1. 

This testimony summarizes findings and recommendations from the independent 
verification and validation (IV&V) review of the VCF Delivery 1, conducted by The 
Aerospace Corporation (Aerospace). This testimony is extracted from Aerospace Re- 
port No. ATR-2005(5154)-4, “Independent Verification and Validation of the Trilogy 
Virtual Case File, Delivery 1: Final Report”, delivered to the FBI on January 21, 
2005. The FBI, and the Vice President, Civil and Commercial Operations of The 
Aerospace Corporation have approved release of this information. 

This overall scope of the IV&V assessments of the VCF Delivery 1 included the 
system design, software design, overall security, and the maturity of the develop- 
ment contractor’s software development processes. Each assessment comprised re- 
views and analyses of pertinent documentation, source code, and process-related ma- 
terials. In addition, the assessment of the maturity of the development contractor’s 
software development processes included a site visit (November 9, 2004) with inter- 
views of key contractor personnel involved in the VCF Delivery 1. The assessments 
summarized in this testimony were conducted in the period August-December 2004. 

It is important to clarify that this effort was not an IV&V in the traditional sense 
of verifying that all requirements have been satisfied, though requirement satisfac- 
tion was part of the assessment. Neither was it an independent program assessment 
that focused on the entire range of management, programmatic, contractual, and 
technical issues. Rather, Aerospace conducted a detailed engineering assessment of 
VCF Delivery 1 requirements and design documentation, source code, and artifacts 
to provide a recommendation to the FBI on discarding or remediating VCF Delivery 
1 products. 

Specifically, Aerospace was asked to address the following business questions: 

Question 1. Did the incumbent contractor meet the stated requirements? 

a. User Needs 

b. System Requirements 

c. Software Requirements 

Question 2. Did the incumbent contractor develop a complete and correct Concept 
of Operations, System Architecture, and System Requirements? 

Question 3. What should the FBI do with VCF Delivery 1? 

a. Keep all of it? 

b. Keep parts of it? 

c. Discard it? 

The remainder of the testimony is organized as follows: 

Section 2 describes the methodology used in assessing the system design associ- 
ated with VCF Delivery 1, as well as the software design, security, and the maturity 
of the development contractor’s software development processes. 

Section 3 summarizes the findings made by the assessment teams in terms of top- 
ics whose state of being influences the answers to the three business questions. 
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These topical groupings represent (1) architecture, (2) requirements, (3) software 
quality, (4) performance, (5) security, and (6) contractor processes. More detailed 
finding statements are found in the Appendices. 

Section 4 presents conclusions formed by examining the findings across all six 
items of interest, as well as inferred findings based on possible observed trends. 
This section addresses Business Questions 1 and 2. 

Section 5 presents a framework for addressing Business Question 3 and a rec- 
ommendation based on the framework. In addition, general recommendations are 
given based on Aerospace observations. 

2. APPROACH 

The IV&V review consisted of assessments of the UAC documentation and arti- 
facts relating to system design, software design, security, and the maturity of the 
development contractor’s software development processes. In addition, the IV&V as- 
sessment of the maturity of contractor processes included a fact-finding trip to the 
contractor’s facility to conduct interviews and view additional materials. In general, 
the methods used were tailored versions of those employed by Aerospace in per- 
forming IV&V reviews of national security space systems. The specific approaches 
utilized by a given assessment team are summarized in the following sections. 

Because IV&V is the process of verifying that requirements are satisfied and vali- 
dating that user needs are met, and because Aerospace was limited primarily to doc- 
umentation and artifacts, most of assessment was spent examining the quality of 
and traceability through the documentation and artifacts. This is in keeping with 
an essential tenet of systems engineering that necessary conditions for a system to 
be successfully implemented are that (1) documentation and artifacts be complete, 
clear, concise, precise, and mutually consistent, and (2) requirements be properly de- 
composed with bi-directional tracing between successive levels of the system (e.g., 
user needs trace to system requirements, system requirements trace to subsystem 
requirements, and so forth through design, implementation, and test). Not only do 
these conditions increase the probability of successfully implementing a system, 
they are required for effective maintenance. 

When possible, the assessment team used industry and government standards as 
benchmarks against which the program documentation and artifacts were meas- 
ured. Although standards were not required on the VCF development contract, 
standards were used in the assessment because they encapsulate known best prac- 
tices that should be used whether or not they are required of a contractor. The use 
of standards also eliminates a level of subjectivity from the assessment. 

Given the scope and time constraints of the IV&V review. Aerospace focused on 
a sample of program documentation and other artifacts. Two notable exceptions 
were that (1) the group assessing the maturity of contractor software development 
processes conducted a 1-day site visit with the contractor to obtain answers to proc- 
ess questions and to view sample reports and artifacts, and (2) a limited number 
of Aerospace personnel attended a 1-day design review. In taking this overall ap- 
proach, it is important to note: 

— With the exception of the 1-day site visit and the 1-day design review. Aero- 
space did not have direct contact with the incumbent contractor to address com- 
ments on the documentation and potentially alleviate some concerns. 

— With the exception of database performance testing, access was not provided to 
the tests that occurred or the results of those tests (hence, the review does not 
directly address how well VCF Delivery 1 satisfies the user requirements but 
does so by inference). 

2. 1 System Design Assessment 

The system design assessment provided the system-level portion of the IV&V re- 
view. The system design assessment was divided into two smaller assessment activi- 
ties: an evaluation of the system-level documentation (i.e., cross-checking the sys- 
tem-level documentation) and a system-level IV&V appraisal of VCF Delivery 1. The 
latter consisted of an examination of requirements traceability, requirements satis- 
faction, performance, and security. 

2.1.1 System Level Documentation Assessment 

To objectively assess the system-level documentation. Aerospace identified stand- 
ards against which the documents could be compared. This section describes the 
ways these standards were used in the assessment. 

The CONOPS was reviewed and its content compared against the reference stand- 
ard embodied in the Department of Defense (DOD) Data Item Description (DID) 
Operational Coneept Description (OCD) [1]. (The emerging guide for preparing 
CONOPS documents [2] that is being created by the American Institute of Aero- 
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nautics and Astronautics (AIAA), in conjunction with the International Council on 
Systems Engineering (INCOSE), was also consulted for content and language.) In 
the review, particular attention was given to the CONOPS with respect to: 

— The description of the current system (e.g., operational environment; major sys- 
tem components; interfaces to external systems or procedures; capabilities and 
functions of the current system; diagrams/charts depicting data flow and proc- 
esses; quality attributes such as reliability, availability, maintainability, flexi- 
bility, extensibility; personnel; support concept for the current system). 

— The justification for and the nature of changes (e.g., description of the needed 
changes; priorities among the changes; changes considered but not included; as- 
sumptions and constraints). 

— The description of the new system. 

— Operational scenarios (e.g., the role of the system and interactions with users; 
events, actions, interactions, stimuli). 

— The new system’s operational and organizational impacts. 

— The analysis of the proposed system (e.g., summary of advantages; summary of 
disadvantages/limitations; alternatives and trade-offs considered). 

The SADD was reviewed and its content compared to the reference standard 
found in the DOD DID System ! Subsystem Design Description (SSDD) [3]. (The In- 
stitute of Electrical and Electronics Engineers (IEEE) Recommended Practice for Ar- 
chitectural Description of Software-Intensive Systems, IEEE Std 1471-2000 [4], was 
consulted for content and language.) The SADD was examined with respect to its: 

— Presentation of system-wide design decisions. Specifically, decisions regarding 
system behavior and the selection and design of components; inputs, outputs, 
and interfaces; actions the system would perform in response to inputs or condi- 
tions; description of physical systems; selected algorithms; how databases would 
appear to the user; approaches to meeting safety, security, and privacy require- 
ments; design and construction choices. 

— Descriptions of the system architectural design (e.g., hardware configuration 
items, computer software configuration items, and manual operations; concept 
of execution; interface design; requirements traceability). 

The SRS was also reviewed and compared against two applicable standards: DOD 
Military (MIL) Standard (STD) 498, Software Development and Documentation [5] 
and DOD DID System ! Subsystem Specification (SSS) [6]. (The INCOSE Systems 
Engineering Handbook [7] was consulted for content.) The SRS was assessed against 
the full breadth of possible requirements, to include: 

— Definition of required states and modes 

— Internal and external interface requirements 

— Internal data requirements 

— Safety requirements 

— Environment requirements 

— Computer-related requirements (e.g., resources, hardware, resource utilization, 
software, computer communications) 

— Quality factors. 

In addition to performing reviews of the SADD, CONOPS, and SRS against par- 
ticular standards, the system-level documentation was assessed for their mutual 
consistency, completeness, and reasonableness. 

2.1.2 VCF Delivery 1 Assessment 
2. 1.2.1 Requirement Traceability 

Aerospace examined the completeness and consistency of user need statements 
and their maturation into system requirements. 

Aerospace extracted all system and software requirements from traceability tables 
found in the SRS and the SRD, and examined parent-child relationships between 
these documents. Comparisons were made of each system requirement statement 
within the body of the SRS to that found in the SRS traceability matrix. A similar 
comparison was made with software requirements in the SRD. 

Validation and verification was performed on subsets of the system-level require- 
ments involving access control and workflow (these requirement areas were chosen, 
in consultation with the FBI, based on their importance to the UAC). Specifically, 
Aerospace identified 22 system-level access control requirements and assessed all of 
them. Of the more than 120 system-level workflow requirements identified, 52 were 
assessed. The 74 system-level access control and workflow requirement statements 
were assessed against the following quality attributes provided in The Engineering 
Design of Systems [8]: 

— 1. Clear and concise . — The requirement has only one interpretation and does 
not contain more than it should. When clarity was in question, the UAC Re- 
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quirements Terms and Definitions Document (RTDD) was used as the primary 
source for clarification. 

2. In-scope. — The requirement does not impose anything unnecessary on the sys- 
tem. 

3. Design- and implementation- free. — The requirement does not impose a design 
or implementation solution. 

4. Verifiable. — The requirement uses concrete terms and measurable quantities. 

5. Free of TBD/TBR. — The requirement does not contain placeholder statements 
or values. 

6. Free of eonfliet or duplication. — The requirement neither overlaps nor opposes 
another requirement. 

7. Appropriate decomposition. — The traced-to software requirements make sense 
and are complete. 

8. Complete requirement set. — There is no appearance of missing requirements re- 
lated to the requirement being examined. 

2. 1.2.2 Requirements Satisfaetion 

Actual requirement satisfaction, as determined through a review of requirement 
testing results, was not considered because test results were not made available. For 
this reason, Aerospace relied on secondary indicators of requirement satisfaction. 
For example, the assessment of traceability of the CONOPS, SADD, and SRS was 
performed within the system-level requirement traceability activity (Section 2. 1.2.1), 
while traceability of software requirements was examined in the software source 
code and traceability analyses (Sections 2.2.3 and 2.2.5). Other facets of requirement 
satisfaction were provided by other analyses. 

2. 1.2.3 Performance 

Contractor test methodology and database performance test results, as found in 
the Interim Scaling and Performance Test Report, were examined to assess the per- 
formance of VCF Delivery 1. The goal of the database performance evaluation was 
to identify areas of high performance risk in the database schema and database 
Structured Query Language (SQL) query code. Network, application server, and web 
server performance were not examined. 

In addition to examining the contractor test data, independent checks on database 
performance were conducted through the following means: 

— Creation of an Entity Relationship diagram based on the contractor database 
Data Definition Language (DDL) code, from which further analysis of the data- 
base could be conducted. 

— Examination of SQL code with respect to (1) system queries, especially with re- 
spect to the use of table joins in clauses, nested queries, outer joins, and 
cursors; (2) code complexity; (3) performance risk factors; and (4) identifying the 
SQL code critical path i. 

— Review of the database structure for signs of performance enhancement at- 
tributes (e.g., table partitioning, table splitting, denormalization, materialized 
views, and rollup tables). 

— Review of the database indexing to determine if table indexes were selected for 
maximum SQL code performance. 

— Analysis of the Virtual Private Database (VPD) implementation performance 
risks (i.e., looking at the where clause predicates that would be added to each 
and every SQL query). 

— Evaluation of system scalability requirements through an extrapolation of re- 
ported test results. 

2.2 Software Design Assessment 

The software design assessment comprised six distinct analyses: software archi- 
tecture, software requirements, source code traceability, source code documentation, 
requirements traceability, and security. 

2.2. 1 Software Architeeture Analysis 

The analysis began with a review of the CONOPS, SADD, SRD, Software Design 
Document (SDD), and accompanying component SDDs. In addition, IEEE Std 1471- 
2000 [4] was reviewed because it was referenced in the SADD. 

The software architecture was examined using an abbreviated form of the Archi- 
tecture and Tradeoff Analysis Method (ATAM) developed by the Software Engineer- 
ing Institute [9]. Critical system and software requirements (known as quality at- 
tribute requirements in the ATAM) were identified in Exhibit 3-2 of the SADD, re- 


1 Critical path SQL code is defined as those SQL queries that are executed a majority of the 
time. 
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viewed, and laid out to form a quality attribute tree, with specification down to the 
scenario level. (These system quality factors address scalability, extensibility, reli- 
ability, performance, security, and evolvability.) Software architectural approaches 
based on the high priority quality factors were then iteratively elicited and ana- 
lyzed, with risks, sensitivity points, and tradeoff points identified. Part of the 
iterative process included brainstorming and prioritizing the scenarios generated in 
the utility tree based on stakeholder needs (in this case, because access to the actual 
stakeholders was not possible, the prioritization was based on information in the 
document artifacts); in the second pass of the process, the scenarios were treated 
as test cases for the architecture analysis. 

2.2.2 Software Requirements Analysis 

Software requirements analysis was conducted on data access control and basic 
workflow requirements after a review of the SRD, SDD (and corresponding vol- 
umes), thread design documents, and consultation with the FBI. The quality of 
these software requirements was evaluated against the following attributes found 
in IEEE STD 830-1998 [10] 2; 

— 1. Unambiguous and clear. — The requirement has only one interpretation. The 
UAC Requirements Terms and Definitions Document (RTDD) was the primary 
source for clarification, followed by Webster’s Dictionary [11]. 

— 2. Consistent. — The requirements do not conflict, and requirements use the 
same terms to mean the same things. 

— 3. Non-redundant. — There are no superfluous requirements. Each requirement 
adds something new to the SRD. 

— 4. Complete. — Nothing is missing from the requirement. Each requirement de- 
fines a user type, employs the verb “shall” once, and specifies an end result. 
Most requirements should also have a performance or timing criterion. 

— 6. Single requirement and concise. — The requirement does not contain more 
than it should. The requirement has no superfluous detail and expresses only 
one need. 

— 6. Design- and implementation-independent. — The requirement does not pre- 
scribe any design or implementation solution. 

— 7. Testable ! verifiable. — The requirement uses concrete terms and measurable 
quantities. Words like “good,” “well,” and “usually” signal that a requirement 
is not testable. 

— 8. Complete requirement set. — No requirements are missing. The set of require- 
ments defines those actions the software will take given all possible types of 
input data when in all possible states. 

Information and findings were shared with and by the system design assessment 
team to increase overall understanding of critical requirements. 

2.2.3 Source Code Traeeability Analysis 

This section summarizes the combined processes of the source code traceability 
analysis and the software requirements traceability analysis (Section 2.2.5). 

Requirements in the areas of access control and basic workflow were identified 
and traced from the software requirements to threads and SDD volumes to the 
source code, using the SDD and corresponding volumes (e.g., Workflow Volume), 
thread design documents. Test Plan, and the RequisitePro® database. (The initial 
process of tracing from software requirements to threads was abandoned after the 
FBI notified Aerospace that the contractor had developed new documentation.) In 
conducting these traceability analyses, emphasis was placed on: 

— Correctness (e.g., does the documented design and source code address the soft- 
ware requirements allocated to it?) 

— Consistency (e.g., is the allocation of software to design and code consistent 
across the documentation and supporting requirements management tools; are 
allocations at the same level of detail?) 

— Completeness (e.g., are all software requirements allocated to design elements 
and code; do the design elements clearly and concisely satisfy the allocated re- 
quirements given the design level of detail?) 

Tracings were examined from software requirements through software design and 
code, and from software requirements to tests. 


2 IEEE STD 830—1998 was used because software requirement specifications and system re- 
quirement specifications are different, and each has a different set of recommended practices. 
There is a lot in common between standards for system requirements and IEEE STD 830—1998, 
and hence duplication, but the two types of standards address different areas of scope for dif- 
ferent audiences, and do so at different levels of detail. 
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2.2.4 Source Code Documentation Analysis 

Java source code complexity was determined for all modules. PL/SQL source code 
complexity was examined for modules related to security, basic workflow, adminis- 
tration, and case management software components. The complexity of modules 
written in Java was determined using McCabeQA®. ClearSQL® was used for mod- 
ules written in PL/SQL. Module size, in terms of source lines of code (SLOG), was 
determined for the respective Java and PL/SQL modules because size is another in- 
dicator of complexity. Those modules with the greatest complexity, size, or relation- 
ship to other modules were then subjected to a peer review: 191 Java modules, from 
the functional areas of data access control, workflow, case management, administra- 
tion, and components, out of 309 high-risk modules; all 667 PL/SQL modules related 
to the functional areas of workflow, security, administration, and case management, 
98 of which were determined to be high risk; and 42 JSP modules in the functional 
areas of workflow, security, administration, and case management, based on size 
and relationship to other JSP modules. The underlying source code of the selected 
modules was compared to contractor documentation (SDD and corresponding vol- 
umes, thread design documents. Software Development Plan (SDP)), especially with 
respect to design and test. Documentation was examined for correctness, consist- 
ency, completeness, and suitability. The Java and PL/SQL peer reviews focused on 
data and control flow, traceability of modules from design documentation, correct- 
ness of comments, and other elements of coding practices as defined by the develop- 
ment contractor’s coding standards expressed in the SDP. 

2.2.5 Requirements Traceability Analysis 

The activities of the source code traceability analysis (Section 2.2.3) and the soft- 
ware requirements traceability analysis were tightly coupled. For that reason, the 
process description and status of the two analyses are combined and reported in 
Section 2.2.3 above. 

2.3 Security Assessment 

The security assessment was based on the DOD Information Technology System 
Certification and Accreditation Process (DITSCAP) [12, 13] and the National Infor- 
mation Assurance Certification and Accreditation Process (NIACAP) [14]. Project 
documentation reviewed as part of the assessment include the SRS, SRD, SADD, 
CONORS, Security CONORS, SDD, Security Volume, Admin Volume, Security Ar- 
chitecture, Security Plan and associated support package. Privileged Users Guide, 
and Certification and Accreditation Methodology. 

Security-related requirements were identified from the available documentation: 
the SRS, SRD, and the Security Volume of the SDD. The design of the system was 
then examined with respect to the subset of requirements to determine the com- 
pleteness and accuracy of the system design against this requirement set. 

Certification and accreditation material contained in the System Security Plan 
and System Security Plan Support Package was also reviewed to determine its suit- 
ability and completeness with respect to what Aerospace experience has shown is 
necessary for such an activity. 

2.4 Software Development Maturity Assessment 

The software development maturity assessment was conducted using the same 
processes Aerospace employs for national security space systems, but tailored to the 
meet the time constraints of this project. A questionnaire was developed, based on 
the U.S. Air Force Software Development Capability Evaluation (SDCE) [15], that 
addresses risks, key requirements, and five areas of specific interest: 

— Systems engineering (e.g., system requirements development, management and 
control) 

— Software engineering (e.g., software requirements management, software de- 
sign, software coding and unit testing, software integration and test) 

— Quality management and product control (e.g., quality management, quality as- 
surance, defect control, peer review, software configuration management) 

— Organizational resources and program support (e.g., organizational process 
management) 

— Program-specific technologies (e.g., database management, COTS, trusted sys- 
tems). 

Answers to some questions were found in a review of the available documentation: 
SRS; Master Plan; Configuration Management (CM), Risk Management (RM), and 
Quality Assurance (QA) Plans; Software Development Plan (SDP); Master Test Plan 
and Delivery 1 Test Plan; and System Security Architecture. Questions that could 
not be answered from the documentation, or for which additional information was 
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needed, were presented to the FBI and the contractor in preparation for an on-site 
fact-finding visit. 

At the time of the fact-finding visit (November 9, 2004) Aerospace interviewed se- 
lected members of the contractor staff according to areas identified in the question- 
naire. The current Deputy Program Director, who was the VCF Delivery 1 Program 
Manager, provided interviewees with the questionnaire and scheduled interviews 
with most of the VCF Delivery 1 managers. The Program Manager accessed the 
IBM Rational® ClearCase®, ClearQuest®, and TestManager® files during the 
interview sessions. Time constraints did not permit an in-depth review of the files, 
but sample reports were printed, and examples of parts of the Software Develop- 
ment Folders (SDFs) were reviewed. 

Prior to the visit. Aerospace requested that the following documents and artifacts 
be available for review: SDFs, the System Engineering Master Plan, documentation 
from preliminary and critical design reviews, deficiency report databases or spread- 
sheets, Rational Rose® artifacts, metrics plans and reports, peer review reports, and 
quality assurance reports. All requested items were made available and reviewed, 
with the following exceptions: 

— The System Engineering Master Plan was not provided. The review team elect- 
ed not to review it because it was not part of the development contract baseline. 

— Rational Rose artifacts were not reviewed. The review team focused on the SDF 
because coding was accomplished based on the SDF contents. 

— No system-level preliminary or critical design reviews materials were reviewed 
because these events were not conducted. Materials from In-Progress Reviews 
(IPRs) and the System Requirements Review were reviewed. 

3. TOPICAL FINDINGS 

The results of the Aerospace IV&V are grouped into six topic areas: 

— Architectures (e.g., enterprise-, system-, and software-level architectures) 

— Requirements (e.g., concept analysis, system analysis, requirement analysis, re- 
quirement quality, traceability) 

— Software quality (e.g., software functionality, structure, testing, documentation, 
thread methodology, database software) 

— Performance (e.g., overall system performance of the database) 

— Security (e.g., certification and accreditation, system security administration, 
security requirements definition, security design documentation) 

— Contractor processes (e.g., processes defined by the contractor that were or were 
not followed, processes that worked or did not work). 

Findings in each area are summarized in the following sections. Each summary 
lists strengths and weaknesses, provides a high-level summary of the most impor- 
tant strengths and weaknesses (individually or in groups) and their implications, 
and gives an overall appraisal of the topic area. Conclusions based on the findings 
are summarized in Section 4. 

With the exception of the software development maturity assessment, all of the 
assessments were made strictly on documentation and artifacts delivered to Aero- 
space. This has two consequences. The first consequence is that this usually leads 
to noting more weaknesses than strengths. If there is sufficient ambiguity or uncer- 
tainty of what is intended in a document, a negative finding is generated, even if 
a short conversation with the contractor could have removed the problem. Therefore, 
the perceived state of what is being evaluated can be more negative than the actual 
state warrants. Aerospace did three things to reduce both the likelihood of this hap- 
pening and the associated impact. First, a fact-finding visit was made to the devel- 
opment contractor’s facility to resolve questions about their software development 
processes. Second, industry and government standards were used to provide objec- 
tive measures of quality and practices. Lastly, Aerospace looked at both documenta- 
tion and product (i.e., source code) for possible strengths or weaknesses in each 
area. 

The second consequence of basing the IV&V review largely on documentation is 
that the ability to transfer the existing document set from the development con- 
tractor to a replacement contractor is tested. In this instance many weaknesses 
could indicate there are significant problems with the documentation or that the 
concepts being developed are not clearly stated. In either case, it would be very un- 
likely that a replacement contractor could pick up where the original left off, there- 
by closing the door on a possible acquisition or maintenance strategy. 

3.1 Architecture 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
the architecture. 
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3.1.1 Strengths 

The incumbent contractor specified a standard three-tiered Web-based design pat- 
tern for the VCF architecture. A well-designed and implemented system of this type 
should be highly flexible, extensible, and scaleable, and should easily integrate new 
functionality. The theoretical strength of the approach is that it is highly 
componentized, generic, and built on open standards. 

3.1.2 Weaknesses 

Though the fundamental strength of the architecture lies in its classic three-tier 
model, the fundamental weakness relates to the failure to actually implement the 
system according to the specified architectural concept. As a result, the system risks 
the ability to maintain, change components (e.g., COTS, GOTS), reuse, or add new 
functionality to the software. Maintainability and reuse are negatively impacted by 
the tightly coupled, threaded design. Performance and scalability are likely to be 
limited by the decision to implement VCF in a centralized versus a distributed fash- 
ion. Furthermore, it is possible that certain types of distributed architectures would 
provide greater reliability through redundancy. 

Though meiximizing the use of COTS was a stated goal of the VCF program. Aero- 
space found a limited use of COTS application products, and a design approach 
whereby functionality available in COTS was rejected, then reimplemented in VCF 
custom code. In addition, no non-Oracle COTS search and analysis tools were found 
to be acceptable, as no non-Oracle tools were found to be compatible with the Vir- 
tual Private Database and associated access controls. 

The use of most COTS software is precluded by the choice for implementation of 
security and access controls at the data level. The VCF system uses two types of 
access controls: functional access controls, and data access controls. Functional ac- 
cess controls are implemented primarily in application code written in PL/SQL with- 
in the data tier. Data access controls are implemented using the VPD. Because all 
of the access control mechanisms are enforced by the database, they cannot be uti- 
lized by external applications. This is a fundamental limitation in VCF architecture. 
This means that virtually all functionality available in COTS that requires access 
control (including document management, workflow, tasking and delegation) must 
be implemented by developers in the VCF application in custom code. This limita- 
tion extends to highly capable COTS search and analysis applications, including 
link analysis and specialized applications used in other law enforcement and intel- 
ligence community applications. 

The manner in which the access controls were implemented in the VPD feature 
of the Oracle database also imposes significant and unacceptable performance 
delays. While most implementations incorporate some of the controls available in 
VPD, and apply to a restricted subset of database tables, this implementation uses 
all of the control mechanisms and applies them to the tables that are used in vir- 
tually every join operation required for the response to any normal database query, 
resulting in significant performance degradation. 

Remediation of these weaknesses would require a complete reevaluation of the ap- 
proach to security access control. 

Lastly, the software architecture documentation does not conform to the best prac- 
tices identified in IEEE Std 1471-2000. For example, stakeholder concerns are not 
directly mapped to the software architectural responses, there is no viewpoint speci- 
fication for the software architecture description, a specific methodology is not iden- 
tified to represent architectural views, and known inconsistencies among architec- 
tural description elements are not noted. Failing to adhere to best practices can im- 
pact functionality, timeliness, and schedule throughout the development cycle. 

3.1.3 Appraisal 

Decisions on architecture and the accompanying high-level design are fundamen- 
tally important. Yet critical architecture goals have not been met. There was a fail- 
ure to appropriately assess the use of COTS products. It appears that inadequate 
attention was given to the performance requirements in relation to the choice of the 
Virtual Private Database and the associated Access Control List (ACL) table to im- 
plement the discretionary access control requirements. Analysis targeted at deter- 
mining the objects to be protected with discretionary access controls, and methods 
of protecting these objects, may have resulted in alternate design choices that had 
more attractive performance characteristics. Likewise, by allowing the original 
three-tier architecture to collapse to two tiers (thus failing to adhere strictly to the 
Web-based design pattern), tbe architectural tenet on separation-of-concerns was 
violated. Consequently, future technology insertion is at risk, and maintenance and 
reuse of the VCF software will be more difficult. 
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3.2 Requirements 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
system and software requirements (development, analysis, and documentation). 

3.2.1 Strengths 

All of the system level requirements examined were found to be within scope. 
There is no evidence of unnecessary features that could constrain design and in- 
crease cost. 

The System Requirements Specification (SRS) did not contain TBD (to be deter- 
mined) or TBR (to be reviewed) markings. This is generally a positive indicator for 
systems that have progressed from the conceptual phase to the development phase, 
since a lack of TBDs and TBRs usually means that the requirements baseline is sta- 
ble. However, the lack of TBDs and TBRs provides no assurance that requirements 
are not missing. Friedman and Sage [16] have pointed out that the lack of TBDs 
can indicate that requirements have been suppressed or ignored, thus creating what 
they call “silent specs.” 

Design and implementation details were not found in either the system-level or 
software requirements; therefore, the developer adhered to expected system and 
software development practices. 

None of the examined data access control and the basic workflow software re- 
quirements duplicated another. Avoiding duplicate requirements eliminates needless 
requirements analysis and redundancies in development and testing. 

3.2.2 Weaknesses 

The CONORS is incomplete in that it lacks summaries of advantages, disadvan- 
tages, limitations, and alternatives and tradeoffs considered. It fails to show through 
analysis that the Information Presentation and Transportation Network Compo- 
nents provide the necessary infrastructure to meet UAC requirements. 

The CONORS does not agree with the SRS, resulting in concepts that are not ar- 
ticulated as requirements in the SRS and requirements that do not correspond to 
operational concepts. The expected relationship between the CONORS and the SRS 
is that the CONOPS should contain statements of operational activities; the SRS 
should specify system functions through functional requirements. A relationship 
should exist between the operational activities and the system functions. Contrast 
this relationship with that between the UAC CONOPS and the UAC SRS: the rela- 
tionship between operational activities and system functions is missing; there is lit- 
tle correspondence between the statements made in the UAC CONOPS and the 
UAC SRS functional requirements. 

Neither the SRS nor the SRD address all of the requirements expected in a speci- 
fication. Failure to address the range of applicable requirements can result in a sys- 
tem that is implemented in such a way as to be unacceptable to the user or other 
stakeholders. Incorporating the additional sections at this point in the life cycle 
would require a major effort that would subsequently result in rewriting the design 
documents and making changes to the source code as needed to accommodate these 
design changes and would result in additional integration and test effort. 

The System Architecture Design Document (SADD) is incomplete relative to ex- 
pectations. Although the SADD lists architecture constraints and goals, it does not 
describe how the architecture meets them. The SADD includes neither decisions nor 
rationale for the external interfaces, scalability, extensibility, maintainability, and 
other items important to the architecture. The incomplete description of the system 
design could lead to unspecified and untraceable software requirements, which, in 
turn, leads to a system that does not meet users’ needs. 

Inconsistencies exist between the Interface Definition Document (IDD), the Inter- 
face Control Documents (ICDs), and the SRS. For example, not all ICDs are ref- 
erenced in the IDD, and some external systems noted in the SRS do not have a cor- 
responding ICD. Although the SRS identifies external systems that currently inter- 
face with ACS and the types of interface to be supported by VCF to ensure legacy 
support, there are no requirements in the SRS that indicate the VCF must ensure 
such support. The IDD itself contains only seven requirements (“shall” statements), 
six of which relate to the frequency of interface execution. Inadequate interface defi- 
nition puts at risk the ability of VCF Delivery 1 to operate with legacy systems. 

In addition to reviewing the requirements-related documentation for inclusion of 
information typically expected in the documents, a quality review of the system-level 
and software requirements was conducted. Quality deficiencies include problems 
such as compound requirements, conflicting requirements, ambiguous and undefined 
terms, use of “and/or” in system requirements, use of “et cetera” in system require- 
ments, use of unverifiable words in system requirements, lack of specified user cat- 
egory in software requirements, lack of response time constraints in software re- 
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quirements, and redundant system requirements. These quality deficiencies may re- 
sult in a system implementation that does not meet the expectations of users and 
other stakeholders. Specific problems and examples are provided in the finding sum- 
maries. 

The SRS did not completely cover requirements. Gaps in expected system level 
functionality were found. Additionally, the Requirements Terms and Definitions 
Document (RTDD) contained implied requirements. Placing implied requirements 
within the RTDD does not ensure that the expected functionality will be imple- 
mented. 

Traceability was assessed on various levels and from various perspectives. The ex- 
pected relationship between need statements, system requirements, and require- 
ments for lower-level elements (e.g., hardware and software) is that there is a strict 
downward flow of requirements. Every need statement maps to a system require- 
ment and every system requirement maps to a need statement. Thus, there are nei- 
ther “childless” need statements nor “orphan” system requirements. The process 
continues in a like manner for the system requirements and lower-level element re- 
quirements. Contrast this with what is observed in the UAC need statements and 
requirement documents: need statements do not flow exclusively to system require- 
ments, and in many cases they bypass the system requirements completely. The 
lack of traceability from need statements to system requirements could result in a 
system design that does not meet user needs and may implement features that are 
not required. 

Traceability was also assessed in the SRS review by conducting a decomposition 
analysis on a set of requirements from the SRS to the software level. The analysis 
identified problems such as incomplete decompositions and decompositions that 
were more restrictive than the system level requirement. Additional traceability 
analyses assessed the mapping of system and software requirements to the 
traceability matrix; errors were found in the trace. The mapping of business rules 
to software requirements was also incomplete. Here again, the lack of traceability 
from system requirements through design means that the design may not meet re- 
quirements and may implement features that are not required. 

Finally, analyses were conducted of traceability from software requirements to 
software design and source code, and from software requirements to tests. There 
were three sets of artifacts that provided traceability between the software require- 
ments and the design: the RequisitePro® database, the thread documents, and the 
SDD volumes. The RequisitePro® database traced to the name of a thread, which 
was associated with the corresponding portion of the SDD volume for basic workflow 
and for data access control. The RequisitePro® database was consistent with the 
traceability in the SDD volumes (with one exception), but not consistent with the 
traceability in the thread documents. It is Aerospace’s understanding that the 
thread documents were the original design documents and that the SDD volumes 
reflected the as-built software. All but one of the software traceability findings deal 
with the SDD volumes as opposed to the thread documents. 

There was poor traceability from the software requirements for basic workflow 
(BW) and data access control (DAC) through the design to the software components. 
A spot-check analysis of the workflow code shows that some software requirements 
do not appear to be covered in the code itself There are some DAC software require- 
ments that are inconsistently traced between the RequisitePro® database and the 
Security Volume of the Software Design Document. Lack of adequate requirement 
traceability into software design and code results in risk that the software will not 
meet its stated requirements and greater difficulty of modifying software when re- 
quirement changes occur. 

There were several BW software requirements that were not assigned to tests in 
the Delivery 1 Test Plan. Without these requirements being validated in assigned 
tests, there is no certainty that the users’ requirements are completely met. 

3.2.3 Appraisal 

The requirements, analysis, and documentation associated with the UAC and 
VCF Delivery 1 contain significant information deficiencies that must be corrected 
to ensure an adequate system definition and development process; a majority of the 
system and software requirements examined contain quality deficiencies; and the re- 
quirements decomposition and traceability chain, from the SRS to SRD to software 
design components to source code to test documents, is weak because of missing in- 
formation and inaccuracies. Extrapolating these observations to the entirety of the 
requirements, analysis, and documentation leads to serious concerns about the 
maintainability and reusability of VCF Delivery 1. Remediation could be very time- 
consuming and, because of the traceability concerns, may not ensure that all prob- 
lems would be addressed. 



49 


3.3 Software Quality 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
the software, to include database software. 

3.3.1 Strengths 

VCF Java Package Standards, set forth in Appendix A of the Software Develop- 
ment Plan, were followed. In particular, the Struts framework was followed, compel- 
ling developers to follow a Model View Controller design. This was significant be- 
cause developers familiar with Model View Controller design and the Struts frame- 
work should be able to understand the flow of information in the source code and 
maintain the presentation layer of software with little difficulty. Also, the software 
components described in the Workflow Volume of the SDD were almost all found 
in the code. This is important for maintenance purposes. 

3.3.2 Weaknesses 

Commenting standards were not consistently followed in the source code files that 
were analyzed. For example, very few functions or files included a change history, 
and there were no references to the design documents associated with each class. 
This inconsistency indicates that coding standards listed in the Software Develop- 
ment Plan were not always followed. Not following a formal software development 
process for such a large system implies the lack of a disciplined approach, a lack 
of coordination among developers, and a lack of standards enforcement. The result 
of inconsistent comments is that the burden of source code maintenance increases 
because programmers are forced to search through the documentation every time 
the code needs chan^ng or when checking for possible side effects associated with 
making changes to different classes. This weakness can be corrected only by going 
through all the source code files and writing the needed comments. There are also 
comments in the code that mention work that remains to be done. This means that 
either the code is incomplete or that the misleading code documentation was never 
removed from completed code. This code should be examined in detail and compared 
to the design to determine its status, and it should be tested to be sure it ran with- 
out errors. Then these comments should be removed to eliminate confusion. 

Some Java classes have modules with incomplete code and unused code. This code 
cannot be validated because its purpose is unknown. Such code can affect the safety 
of the system by performing unexpected and unplanned operations. If the code is 
not fully validated, then the proper operation of the system cannot be assured. 

Some Java code contains inconsistent use of constants by hard coding and some 
by using constants and database files for others. The preferred method is to use con- 
stants and database files so that any future changes can be made to the constant 
or to the database, thereby ensuring completeness of the change. Hard coding re- 
quires that changes be made to all instances and some may be missed. 

Discrepancies were found between the thread design documents and the Software 
Design Document volumes for data access control and basic workflow. In addition, 
there were cases where more detailed design was found in the thread documents 
than in the design documents. Inconsistent design documentation is confusing to 
anyone trying to understand, maintain, or modify the software. 

The design documentation reviewed does not bridge the gap between the Software 
Design Document volumes and the source code. Missing design information included 
the relationships between the software components; class parameter details; full def- 
inition of class interfaces; and details on the purpose and logic of each function. The 
existence of the code is listed in the high-level design, but not the code behavior and 
interactions, which should be reflected in the lower level, detailed design. Examples 
of missing design details include: (1) the PL/SQL code for workflow contained a total 
of 111 modules, of which 57 were not mentioned in the SDD; (2) a discrepancy be- 
tween the Java files listed in the SDD volumes and the source code provided to 
Aerospace. The lack of a detailed design document that included all source code 
modules makes maintenance and modifications to the source code more difficult and 
time-consuming, and would subsequently drive up the cost of any future changes to 
the system. 

The PL/SQL code has timing and design issues. With regard to timing, each mod- 
ule writes a character string to the debug log (in one module printing is initiated 
through the use of a debugging switch; in all other cases the printing is hard coded). 
This has a negative impact on code performance because it increases execution time; 
this practice would be tolerable only during prototype development. As to design, 
the PL/SQL code uses literals rather than symbolic constant variables in the argu- 
ments of “IF” and “WHERE” statements. This code would be nearly impossible to 
maintain by anyone other than the programmer who developed it because there are 
no references to design documents that define literals, such as the integer-type val- 
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ues. This is an example of not following coding standards, or of not enforcing them. 
It would take time to fix this code properly by replacing the literals with s 3 unbohc 
constant variables so that it could be understood by anyone other than the original 
programmer. 

3.3.3 Appraisal 

The source code appears to have been produced without adherence to the proce- 
dures and standards stipulated in the Software Development Plan. The source code 
examined is not maintainable with its current documentation. Without reverse-engi- 
neering the missing documentation and conducting thorough testing, the code 
should not be used for any operational system. To reverse-engineer this system to 
bring it up to the level for proper maintenance and support would cost about one- 
quarter to one-half of the original cost of development. In most systems, 15 percent 
of the cost is derived from the documentation across all development stages. Testing, 
including documentation, typically accounts for 40 percent. Approximately half of 
the documentation needs to be completed. The remaining testing is expected to be 
between half and all of the cost of a typical system. This depends on problems found 
during testing. 

3.4 Performance 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
system and database performance. 

3.4.1 Strengths 

None identified. 

3.4.2 Weaknesses 

The VCF system that was tested by the contractor was a development version 
(VCF Delivery 1), which is missing requirements and is inadequate for operations. 
The system did not implement a number of features, such as the Virtual Private 
Database (VPD) or full production scale of hundreds of millions of rows in the data- 
base. The measurements (as documented in the Interim Scaling and Performance 
Test Report) only present some CPU utilizations and end-to-end response times 
from/to the web server. Disk, bus, and network actual performance were not pro- 
vided in the performance report provided to us and are presumed not to have been 
tested. 

The reported system performance and its performance analysis approach are at 
best marginal. Only the least-complex transactions were reported, and a number of 
those did not meet requirements even for the scaled-down database without VPD. 
A fully populated production VCF system based on VCF Delivery 1 would not meet 
requirements. In some cases the response time would be hundreds of percent longer 
than is required, and in worst cases thousands of percent longer than is required. 
Such long response times are essentially nonresponsive. 

The VCF database has the attributes of a logical database model with large num- 
bers of tables, a lack of denormalization, subtype entities modeled directly to phys- 
ical tables, and other logical data model features. Logical models are rarely perform- 
ance-optimal. The typical database objects available for performance optimization 
(e.g., performance-based index selection, materialized views, table partitioning) are 
absent from the VCF database. At the current estimated row counts, the database 
will require heavy optimization in order to scale properly; however, the developers 
did not do this. 

The database load estimates were created using historical ACS usage. While using 
historical ACS usage was a good starting point, a more thorough analysis of the 
probable usage of VCF should have been performed before translating these esti- 
mates into a testing protocol. 

The database SQL code is not performance-optimized. The SQL code throughout 
the system uses many of the constructs that are specifically noted in the database 
manufacturer documentation [17, 18] as being performance risks. Compounding the 
problem is the use of the Oracle VPD feature for database security. The VCF imple- 
mentation of this feature causes even more poor-performing SQL code to be added 
to each and every SQL statement in the database. 

The executed performance tests were flawed in two ways. First, the contractor did 
not isolate the database when the CPU utilization was tested. Aerospace was unable 
to conclude whether the database CPU was underutilized because it was not having 
a problem with servicing requests or it was waiting on another dependent system. 
The database CPU could also have been waiting on internal database hardware 
such as bus data transfers. The second flaw in the performance tests is that the 
test database was loaded at substantially lower row counts than what is estimated 
as needed, even for the ACS database migration. 
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An enterprise such as the VCF requires to he managed hy a Network Operations 
Center (NOC). The NOC would include a Network Management System (NMS), 
archiving, availability monitoring, and other system and operational functions. 

The NMS would include functions such as a help desk, trouble ticket system, a 
network management console, and network management agents on the managed 
workstations and servers. The NMS would use protocols such as SNMP (Simple 
Network Management Protocol), RMON (Remote Monitoring), and others. The lack 
of a requirement for an NMS would result in a system that cannot be operated in 
a production environment, especially after it is fully scaled to global production size. 
Any production system requires routine archiving. Most systems have an incre- 
mental or even a full backup daily, and a full backup at least weekly. Without 
archiving, work could be lost, evidence misplaced or destroyed, and investigations 
could lose their integrity. The lack of a requirement for archiving would result in 
a system that cannot be operated in a production environment, especially after it 
is fully scaled to global production size. 

Any production system must meet availability requirements commensurate with 
its mission. A system that is unavailable could result in an interrupted investigation 
due to lack of access to investigation data, or the inability to record new information 
that is crucial to progress in the current investigation and other affected investiga- 
tions that depend on new evidence collected. On top of that, investigation resources 
would be lost when the system is down. The lack of a requirement for system avail- 
ability would result in a system that cannot be operated in a production environ- 
ment, especially after it is fully scaled to global production size. 

3.4.3 Appraisal 

The system falls short of meeting requirements as tested. In addition, the scaled- 
up system, with the VPD running, is highly unlikely to meet requirements, particu- 
larly for the type of complex queries needed by VCF. Simple queries would be hun- 
dreds of percent slower than the type of queries that were tested by the incumbent 
contractor. The situation would be far worse for complex queries running on the 
scaled-up system. The system would fall short of requirements with extremely long 
response times — thousands of percent longer than is required. Such long response 
times are essentially nonresponsive. 

The database has many characteristics of a database still in development: a phys- 
ical implementation of the logical database model that will undergo significant 
modification well before production, and SQL coding statements structured in a way 
that is logically sound and easily understood, but not optimized for performance. De- 
velopers typically develop code in this manner, expecting that time will be allocated 
to performance optimization once the code is functionally correct. Code modifications 
are also easier before optimization. 

The database hardware selection appears adequate for the raw amounts of data 
that must be processed, but the database subsystem requires a realistic test with 
all features active, especially the VPD security and a full ACS migration data load. 
The production hardware and COTS software (i.e., Oracle database. Sun server, and 
the Hitachi Storage Area Network (SAN)) are technically capable products. How- 
ever, the current VCF database schema and SQL code implementation do not con- 
tain the performance enhancements that would allow the hardware and COTS data- 
base server to perform optimally. 

3.5 Security 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
security. 

3.5.1 Strengths 

It appears that planning for system security was done at a high level early in the 
program. Such planning increases the likelihood that required security features 
(e.g., access control, audit) will be addressed in the requirements and design, which, 
in turn, provides a cost-effective path to certification and accreditation. Select areas 
of the system not generally found in initial system security reviews (e.g., infrastruc- 
ture devices such as routers and switches that nonetheless contain functionality 
that must be addressed from a security perspective) were addressed in some amount 
of detail. 

The system design provides for a limited interface controlled by the VCF applica- 
tion and infrastructure (for non-administrative users to interact with the VCF). This 
approach prevents exposure to security vulnerabilities that may exist in the inter- 
faces provided by underl3dng products (not visible at the user interface), such as the 
command line for an underlying operating system. 

At a high level, these strengths point to an approach that, if followed, would 
produce an accreditable system. 
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3.5.2 Weaknesses 

Several weaknesses were discovered that create a significant risk that the system 
will not be accreditable. 

Broad areas of security requirements were neither well-defined nor correctly de- 
composed to lower-level requirements. Although the coverage area of the lower-tier 
requirements was the same as that in the higher-level documents, the lower-tier re- 
quirement did not provide the necessary detail to implement and test the system 
in support of the certification and accreitation effort. Furthermore, the documents 
identified as the primary means for the certification and accreditation effort (the 
System Security Plan and the System Security Plan Support Package) did not map 
to the requirements specified for the system. This failure to identify the require- 
ments to which the system would be accredited greatly increases the risk that the 
system would not receive accreditation, even if built to the requirements specified. 
Weaknesses were found in design and implementation. The Privileged User Guide 
should contain information to manage and configure the system in a secure manner. 
However, there are many sections marked TBD, as well as sections that do not pro- 
vide the detailed procedures required to perform critical configuration steps (e.g., 
specific configuration instructions for the boundary devices so that fundamental as- 
sumptions noted in higher-level documents can be achieved). Some of the detail pro- 
vided in this guide also appears as if it were copied from other sources, and not 
modified for application to the VCF system. Without specific configuration informa- 
tion, the trustworthiness of the system cannot be assessed and the system will not 
be accreditable. Furthermore, if the security features that are needed do not exist, 
or do not support all of the capability being depended upon by the architecture, then 
significant schedule and dollar costs will be incurred. 

The design documentation for the audit subsystem does not describe how the 
audit requirements are being met, especially in the area of management of the audit 
trail, while the Privileged User Guide contains COTS audit configuration steps, 
there is no discussion concerning how the VCF audit is managed, and how the VCF 
audit can be integrated with the audit trails produced by the COTS products to pro- 
vide a coherent audit trail. 

3.5.3 Appraisal 

At a high level, the system security description appears to be a good start in de- 
scribing the functionality necessary to build an accreditable system. However, in 
specifying and designing the system to meet that functionality, it appears there are 
significant shortfalls. Select requirements specifying the functionality are imprecise 
and incorrectly decomposed. The design of critical identification and authentication 
and audit subsystems do not implement a significant portion of the requirements 
for those subsystems. The documents supporting the certification and accreditation 
of the system and security configuration are not complete. 

While all of these issues can be remedied, at this point in the product lifecycle 
there is a high risk that the system implementation will not meet the security re- 
quirements, and that significant additional costs (both to the schedule and in dollars 
spent) will be incurred in trying to address the issues identified. There is a high 
likelihood that the system as it currently stands will not be able to be accredited 
without significant additional effort on the part of the developer. 

3.6 Contractor Processes 

This section summarizes the strengths, weaknesses, and Aerospace assessment of 
the contractor processes. 

3.6.1 Strengths 

Strengths regarding the development contractor’s initial processes include: 

— Good organizational structure for program management and quality assurance 

— Selection of requirements, software, and documentation control tools 

— Use of peer review and audits as key elements of the quality assurance process 

— Good configuration management and integrated management tools 

— Tracking of change requests. 

A Chief Engineer was designated to monitor the development and integration of 
the systems engineering, software engineering, and data engineering activities. The 
Quality Assurance Manager reported to the Group-level QA at a level above the Pro- 
gram Manager to provide independent quality assessments of compliance with the 
established procedures. 

Processes and procedures for the software development were defined and docu- 
mented in the Software Development Plan. COTS tools for managing these proc- 
esses were selected and are the same as those used frequently in other government 
developments. Configuration Management to control and track the baselines and 
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changes to the requirements, documentation, and software was defined and con- 
trolled via an integrated, automated tool suite. 

3.6.2 Weaknesses 

The Master Plan did not include planning information (such as key events and 
tasks) and controls (such as system level reviews) for the development task. The im- 
plemented risk management process included only an ad hoc risk identification 
method — personnel identified perceived risks to the Risk Management Board for 
analysis. Although the organizational structure provided for integration of the engi- 
neering tasks, there was a lack of engineering discipline as evidenced by the lack 
of adherence to established processes. 

The software methodology did not provide for the database design, implementa- 
tion, and test. There were neither top-level software descriptions nor interfaces de- 
picted in the Software Development Plan. The database was developed after the 
software design, which led to performance problems. Software integration testing 
was not planned for in the Software Development Plan, and the test plans called 
this by different names without describing how it was to be done. The system inte- 
gration manager and team did the software integration testing, but this was not 
made clear in the documentation. 

Requirements were tracked and reported in the RequisitePro® tool, but software 
requirements were not traced to the code — only to the threads (which is at a very 
high level). 

The quality assurance (QA) program did not include QA activities for the software 
code; QA only checked that the peer review process was followed. 

Software development folder guidelines were published in the SDP and in the 
Minimum Thread Team SDF Layout and Contents, but did not provide for a conven- 
ient structure to maintain the artifacts. SDF files were dispersed in several different 
tools and in many folders, making it difficult to find a complete SDF. 

3.6.3 Appraisal 

The major process strength of VCF Delivery 1 was the documenting and planning 
for the guidelines, procedures, and process controls in the beginning of the program 
in the Software Development Plan. The major weakness was a lack of compliance 
and completeness in the procedures. 

The SDFs were used to maintain the updated requirements analysis, design mate- 
rials, implementation artifacts, testing results, and lessons learned. The SDFs were 
to be the key documentation since the other documentation was not updated. How- 
ever, assessing the completeness of the SDFs is extremely difficult and cannot be 
done without detailed guidance from a developer. 

A major defect for the maintainability, reusability, expandability, and reliability 
of the VCF Delivery 1 software is the lack of a defined and documented software 
architecture and software methodology. Without the tracking of requirements to the 
software, the reliability and usability of the system is questionable and the software 
cannot be verified and validated. Without good software architecture, there is no 
structure to build for future development or functionality. 

4. CONCLUSIONS 

The principal conclusion of the IV&V effort is that a lack of effective engineering 
discipline has led to inadequate specification, design, and development of VCF De- 
livery 1. Most of the findings presented in Section 3 relate in some way to this con- 
clusion. From the documents that define the UAC system at the highest level, down 
through the software design and into the source code itself. Aerospace discovered 
evidence of incompleteness, lack of follow-through, failure to optimize, and missing 
documentation. 

The engineering practices followed on this program were not in keeping with what 
Aerospace would expect in a program of this magnitude and importance. Good engi- 
neering practice includes, of course, well-written requirements that specify the es- 
sential functionality, performance, and constraints of the system. It also includes 

— Modeling to analyze behavior and performance, and to ensure the correctness, 
completeness, consistency, and realism of the requirements 

— A correct decomposition and flow down of requirements from user needs to sys- 
tem requirements to design. 

These practices were found lacking or ineffective for the VCF program. Without 
them there can be little assurance as to the correctness and completeness of the re- 
quirements and design. 

Secondary conclusions address two of the FBI business questions stated in the In- 
troduction. Business Question 2 asks, “Did the incumbent contractor develop a com- 
plete and correct Concept of Operations, System Architecture, and System Require- 
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ments?” and speaks to the framework of the UAC system and VCF Delivery 1. Busi- 
ness Question 1, on the other hand, asks, “Did the incumbent contractor meet the 
stated requirements?” and must be considered in light of user needs, system re- 
quirements, and software requirements. Responses to these questions are not given 
in terms of a simple “yes” or “no,” but are phrased in terms of assurance of an af- 
firmative answer. 

The secondary conclusions, together with their basis in the findings, are given 
below, as well as the summary assessment of the state of the UAC and VCF Deliv- 
ery 1. 

4.1 Regarding the Quality of the CONORS, Architecture, and Requirements 

Findings in the areas of architecture and requirements indicate that the concept 

of operations, system architecture, and system requirements were not sufficiently 
mature for the purposes of developing VCF. The SRS does not provide an adequate 
basis for the developer to design the system. The SRS and the CONORS taken to- 
gether do not provide a complete and consistent view of the system. The SADD con- 
tains certain sound architectural concepts but fails to adequately consider the use 
of alternate architectural concepts or the use of COTS that may have better served 
the needs of the VCF system. Therefore, Aerospace finds no assurance that the ar- 
chitecture, CONORS, and requirements are correct and complete, and no assurance 
that they can be made so without substantial rework. 

4.2 Regarding the Satisfaction of Requirements 

Findings on user, system, and software requirements touch most of the areas of 
interest (i.e., architecture, requirements, software quality, security, and perform- 
ance), and tend to be negative. Based on the requirements examined, the findings 
indicate that a substantial body of requirements are imprecisely written or incor- 
rectly decomposed into lower-level requirements, detailed designs, or test scenarios. 
There are key requirements whose correctness is questionable, and there are nota- 
ble instances where the design and implementation do not match the architecture 
and requirements. The extent of requirement satisfaction could not be fully deter- 
mined because only high-level test plans, software problem reports (SRRs), and a 
performance test report were available; other documents that are normally exam- 
ined in determining requirement satisfaction (e.g., requirement test plans and pro- 
cedures, and results from testing) were not available. There is no evidence that the 
system will scale to the storage and throughput capabilities under the demands of 
a fully loaded scenario; rather, evidence was found to the contrary. Therefore, Aero- 
space finds no assurance that requirements, at the system or software level, will be 
fully satisfied, and no assurance that they can be satisfied without substantial re- 
work. 

4.3 Overall Assessment 

The UAC and VCF Delivery 1 do not adequately meet system and software re- 
quirements. Each of the six areas examined has significant weaknesses and few 
compensating strengths. For example, 

— The architecture was developed without an adequate assessment of alternatives 
and conformance to various architectural standards, in a way that precluded 
the incorporation of significant commercial-off-the-shelf software, and without 
modeling and simulation to determine whether the architecture would meet 
user needs in realistic situations. 

— High-level documents were neither complete nor consistent, and did not map to 
user needs. 

— Requirements and design documents are incomplete and imprecise, requirement 
and design tracings have gaps, and software cannot be maintained without dif- 
ficulty, and is therefore unfit for reuse. 

In short, VCF Delivery 1 is a system whose true capability is unknown and may 
be unknowable, unless substantial time and resources are applied to remediation. 

5. RECOMMENDATIONS 

This section presents a framework for addressing one of the FBI business ques- 
tions set forth in the Introduction and recommendations based on the framework. 
Additional recommendations beyond the scope of the original business questions are 
also provided. 

5.1 A Framework for Addressing FBI Business Question 3 

FBI Business Question 3 asks, “What should the FBI do with VCF Delivery 1?” 
The possible outcomes include keeping all of it, keeping parts of it, or discarding 
all of it. Although this independent assessment is primarily technical in nature. 
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stakeholder interests that affect the disposition of VCF Delivery 1 may be technical, 
budgetary, schedule-based, or mission-oriented in nature. Only the FBI — in consid- 
eration of the various stakeholder interests — can make the ultimate decision on the 
disposition of VCF Delivery 1. 

The decision to be made about VCF Delivery 1 is framed by the conditions that 
must be met in each possible outcome. As understood by Aerospace, the outcomes 
and conditions are as follows: 

— Under what conditions should the FBI keep VCF Delivery 1? 

— Only if VCF Delivery 1 satisfies all requirements or remediation is readily 
achieved. 

— Only if the FBI desires a custom UAC solution versus a solution based on 
COTS software. 

— Only if it satisfies the needs of the FBI with respect to functionality, schedule, 
affordability and life-cycle issues. 

— Under what conditions should the FBI keep parts of VCF Delivery 1? 

— Only if there are separable components of VCF Delivery 1 that contain useful 
functionality in the future context of the UAC. 

— Only if VCF Delivery 1 meets the conditions for reusable or maintainable soft- 
ware. 

— Only if the FBI still desires a custom VCF solution versus a solution based 
on COTS software. 

— Under what conditions should the FBI discard VCF Delivery 1? 

— Only if VCF Delivery 1 satisfies none of the conditions above. 

5.1.1 Regarding the First Possible Outcome 

Regarding the first outcome, keeping all of VCF Delivery 1, Aerospace has no as- 
surance that the VCF Delivery 1 requirements have been met or that remediation 
may be readily achieved. In fact. Aerospace concludes that determining which re- 
quirements are actually met and remediating those that are not would be very cost- 
ly and time-consuming, given that there are serious concerns with every level of the 
system, from the requirements and architecture, to the design and the software. 

5.1.2 Regarding the Second Possible Outcome 

The second outcome, keeping parts of VCF Delivery 1, depends on whether compo- 
nents of VCF Delivery 1 will be useful in some future context. In the current con- 
text, VCF Delivery 1 is custom software based on a centralized hardware architec- 
ture. Thus, if the future context is a COTS-based service-oriented architecture 
(SOA) solution based on a distributed hardware architecture, it is less likely that 
useful components of VCF Delivery 1 will be found. On the other hand, if the future 
context is another centralized hardware architecture with custom software, it is 
more likely that useful components will be found. This last instance is precisely the 
context in which the incumbent contractor is developing the IOC software — and is, 
in fact, reusing components of VCF Delivery 1. 

Even if a future context occurs in which components of VCF Delivery 1 are 
deemed useful. Aerospace has concerns on the reusability and maintainability of the 
software based on the documentation, design, and coding standards. The software 
was not written for reuse and has serious maintainability and extensibility prob- 
lems as well. 

Because the Aerospace IV&V review was based largely on documentation and ar- 
tifacts, and included no substantive direct contact with the development contractor 
other than that needed to assess the software development processes, the ability to 
transfer the existing document set from the development contractor to a replace- 
ment contractor was tested. The many documentation weaknesses that were found 
indicate the existence of significant problems. It is very unlikely that a follow-on 
VCF contractor could pick up where the incumbent left off, thereby weighing against 
this as a possible acquisition strategy. 

5.1.3 Regarding the Third Possible Outeome 

The third outcome, discarding all of VCF Delivery 1, is essentially the default con- 
dition that will occur if none of the preceding conditions are met. It can be reached 
if the VCF Delivery 1 software is found unsuitable for reuse and beyond remedi- 
ation. Alternately, it can be reached by fiat if the FBI should decide — based on the 
results of the Aerospace COTS/GOTS survey [19] — to proceed with a COTS-based 
solution. 

5.1.4 Reeommendation 

It is evident from this decision framework that the VCF Delivery 1 decision de- 
pends on more than just VCF Delivery 1 itself. It depends on the total future con- 
text in which the VCF application will exist. 
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It is clear that the first outcome (keeping or remediating VCF Delivery 1) is not 
feasible because of the lack of assurance that VCF Delivery 1 fully satisfies the sys- 
tem and software requirements, and because Aerospace can foresee no condition 
under which remediation would be feasible. Put another way, any remediation of 
VCF Delivery 1 would be akin to starting over. 

The second outcome (keeping parts of VCF Delivery 1) is more feasible than the 
first, but is still fraught with difficulty. Because VCF Delivery 1 documentation and 
source code do not meet the conditions for reusable or maintainable software. Aero- 
space believes it would be difficult for any contractor (including the incumbent) to 
extract much of value from the current requirements, design and software given the 
poor state of the documentation. Furthermore, Aerospace believes it would be ex- 
tremely difficult for any contractor besides the incumbent to do so. 

Additionally, using VCF Delivery 1 or a derivative thereof only makes sense ab- 
sent a preference for a COTS-based VCF solution. Given that there are multiple 
COTS applications, or features within applications, that meet the needs stated in 
the Federal Investigative Case Management Request for Information (RFI) [19], the 
question of reusing parts of VCF Delivery 1 rests on: 

— Having functionality that is superior to the COTS options or that is not avail- 
able in COTS; 

— Having functionality that is modular and has a defined interface application 
programming interface (API); 

— Its ability to provide a clearly defined service or set of services in the context 
of an SOA. Both the RFI and current federal information technology acquisition 
guidelines (Clinger-Cohen Act [20], Federal Enterprise Architecture Guidelines 
[21]) stress the desirability of SOAs. 

While the discussion to this point is implicitly about the best long-term VCF solu- 
tion, it is also worth considering what may be a useful short-term solution. It may, 
for instance, be the case that the work currently being performed by the incumbent 
contractor on an IOC build will provide a short-term capability to satisfy mission 
needs in a timely fashion until a solution can be crafted that is both more capable 
and more feasible for the long term. Whether or not this is feasible depends on the 
timeliness and affordability and short-term utility of an IOC-like solution versus the 
timeliness of a COTS-based solution (which is a strong contender for the preferred 
long-term solution). 

Thus, pending the outcome of the trade studies recommended below. Aerospace 
believes that discarding VCF Delivery 1 and starting over with a COTS-based solu- 
tion is the best long-term solution. Although Aerospace recommends that VCF De- 
livery 1 not be used as a software baseline for any future VCF activities based on 
the deficiencies identified herein. Aerospace recommends that the VCF Delivery 1 
artifacts (both documentation and source code) and this report be made available 
as Government Furnished Information ^ (GFI) to any future VCF vendors (as part 
of a “Bidders’ Library” for instance). There are insights to be gained from under- 
standing how the VCF problem was initially framed, how the architecture was con- 
ceptualized, and how the system was designed and implemented that Aerospace be- 
lieves will be of use to future developers. Aerospace recommends, however, that 
these artifacts be made available only if accompanied by this report. Otherwise, the 
future vendors will be in the position of having to repeat all the investigation and 
analysis performed by Aerospace in its investigation of those artifacts. 

Based on the RFI responses, there are multiple COTS applications, or features 
within applications, that meet both the SOA requirements and the needs stated in 
the RFI. 

5.2 Additional Recommendations 

The fact that Business Question 3 was asked at all implies that the future of VCF 
is being considered. The larger issue, then, beyond the disposition of VCF Delivery 
1 is the way ahead for VCF. It is in consideration of this larger issue that the fol- 
lowing recommendations are offered. 

The principal conclusion of this assessment relates to a lack of engineering dis- 
cipline and all its negative effects. Accordingly, this problem must be remedied be- 
fore going forward. Broadly speaking, this will require that the FBI specify that ap- 
propriate systems engineering and software engineering practices be defined and 
used, and then provide oversight to ensure that they are followed. Allowance must 
be made for a reasonable schedule. An assessment conclusion states: “The engineer- 
ing practices followed on this program were not in keeping with what one would 
expect in a program of this magnitude and importance.” The specific recommenda- 


3 Providing the documents and artifacts as Government Furnished Equipment (GFE) is not 
recommended so as to avoid the government incurring any liability for their use. 
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tions offered below speak to practices that Aerospace believes are in keeping with 
a program of this magnitude and importance. 

5.2. 1 Concerning the VCF Architecture 

Aerospace found that VCF Delivery 1 began with certain sound architectural con- 
cepts, but failed to consider the use of alternate architectural concepts or the use 
of COTS components that may have better served the needs of the VCF system. 
Therefore, Aerospace recommends that trade studies be performed across several 
key dimensions, to include the following at a minimum: 

— The use of COTS components for key integrated case management functionality 
(not merely for infrastructure items such as databases, operating systems, and 
communications protocols) versus the use of custom application software. 

— The use of an SOA versus the use of a monolithic software application. 

— The use of a distributed hardware architecture versus the use of a centralized 
hardware architecture. 

Additionally, Aerospace recommends an analysis of how VCF fits in with, and is 
constrained by, the broader enterprise architecture. 

5.2.2 Concerning the VCF Requirements 

Aerospace found the VCF Delivery 1 concept of operations and the system require- 
ments to be insufficiently mature for the purposes of the UAC acquisition. There- 
fore, Aerospace recommends that a new series of meetings be conducted with the 
users and other stakeholders to elicit needs, constraints, operational concepts, and 
requirements. Once a set of abstracted needs, constraints, and broader enterprise 
concerns is in place, it will be possible to perform the operational and requirements 
analyses, modeling of operations and functions, and modeling of performance nec- 
essary for the creation of a correct and complete CONOPS and System Require- 
ments Specification. 

Aerospace found a lack of accurate and complete traceability between the various 
levels of requirements, components, and tests. Therefore, in order to provide assur- 
ance that all VCF requirements have been met and verified. Aerospace recommends 
that the any future VCF development and acquisition activities enforce strict 
traceability. 

5.2.3 Concerning the Use of Standards 

Many of the problems with the body of VCF documentation extend beyond a sim- 
ple lack of discipline and instead relate to a failure to address certain standard con- 
cerns in system architecture, system specification, system design, and requirements 
quality. The systems engineering field is sufficiently mature that there are stand- 
ards and other references that provide descriptive outlines for key documents and 
quality attributes for written requirements. 

Many — though not all — of these standards originate in the defense arena. How- 
ever, they are applicable (with tailoring) to non-defense systems such as VCF pre- 
cisely because it is similar to many defense systems in its complexity, its scope, and 
the criticality of its mission. As such, the approaches used in creating other large, 
complex, mission-critical systems can be applied here. The standards and other ref- 
erences that Aerospace applied to the VCF assessment are given in the bibliography 
contained in this document. Aerospace believes they are as applicable to the future 
of VCF as they were to an assessment of its past. 

5.2.4 Concerning Processes 

The success of acquisition programs, particularly large ones, depends not only on 
what is done but also on how it is done. Products result from processes — and it is 
precisely for this reason that processes are important. While a good process is not 
sufficient to produce an excellent product, it is necessary. 

A project of the scope, complexity, and importance of VCF demands the level of 
process maturity embodied in CMMI Levels 3 and 4. CMMI Level 1 and Level 2 
are too “ad hoc” for a program of this nature; on the other hand, CMMI Level 5 
is probably not warranted. 

Aerospace recommends that a Software Development Capability Evaluation be 
conducted prior to contract award to reduce acquisition risk by objectively assessing 
each offeror’s ability to successfully develop the software needed by the VCF pro- 
gram. Aerospace recommends that an independent government cost analysis be con- 
ducted during source selection to objectively assess the cost realism of each offeror’s 
proposal. 



58 


REFERENCE 

[1] U.S. Department of Defense. Operations Concept Description (OCD), Data Item 
Description DI-IPSC-81430, December 5, 1994. 

[2] American Institute of Aeronautics and Astronautics, Guide for the Preparation 
of Operational Concept Documents, Working Draft 1.0, ANSI/AIAA Gr-043-200x. 
Washington, D.C.: American Institute of Aeronautics and Astronautics. 

[3] U.S. Department of Defense. System ! Subsystem Design Description (SSDD), 
Data Item Description DI-IPSC-81432, December 5, 1994. 

[4] Institute of Electrical and Electronics Engineers. IEEE Recommended Practice 
for Architectural Description of Software-Intensive Systems, IEEE STD 1471-2000. 
New York: Institute of Electrical and Electronics Engineers, 2000. 

[5] U.S. Department of Defense. Software Development and Documentation, MIL- 
STD-498, December 5, 1994. 

[6] U.S. Department of Defense. System ! Subsystem Specification (SSS), Data 
Item Description (DID) DI-IPSC-81431, December 5, 1994. 

[7] International Council on Systems Engineering. International Council on Sys- 
tems Engineering (INCOSE) Systems Engineering Handbook, Version 2.0, July 2000. 

[8] Buede, Dennis M. The Engineering Design of Systems. New York: Wiley, 2000. 

[9] Bergey, J. et al. Software Architecture Evaluation with ATAM^’^ in the DOD 
System Acquisition Context. Carnegie Mellon University/Software Engineering Insti- 
tute Technical Note CMU/SEI-99-TN012, ADA377450. Pittsburgh: Carnegie Mellon 
University/Software Engineering Institute, 1999. 

[10] Institute of Electrical and Electronics Engineers. IEEE Recommended Prac- 
tice for Software Requirements Specification, IEEE STD 830-1998. New York: Insti- 
tute of Electrical and Electronics Engineers, 1998. 

[11] Webster’s Ninth New Collegiate Dictionary. Springfield, Massachusetts: 
Merriam-Webster, Inc., 1988. 

[12] U.S. Department of Defense, DOD Information Technology System Certifi- 
cation and Accreditation Process, DOD Instruction 5200.40, December 30, 1997. 

[13] U.S. Department of Defense, Department of Defense Information Technology 
System Certification and Accreditation Process (DITSCAP): Application Manual, 
DOD 8510.1-M, July 31, 2000. 

[14] National Security Telecommunications and Information Systems Security 
Committee. National Information Assurance Certification and Accreditation Process 
(NIACAP) , National Security Telecommunications and Information Systems Security 
Instruction (NTISSI) No. 1000, April 2000. 

[15] U.S. Department of the Air Force, Air Force Material Command. Software De- 
velopment Capability Evaluation, AFMC) Pamphlet 63-103, vols. 1 and 2, June 15, 
1994. 

[16] Friedman, George, and Andrew P. Sage. “Case Studies of Systems Engineer- 
ing and Management in Systems Acquisition,” Systems Engineering, volume 7, no. 
1, 2004, p. 90. 

[17] Green, Connie. OraeleQi Database Performance Tuning Guide and Reference, 
Release 2 (9.2). Oracle Corporation (Redwood Shores, CA 94065), 2002. 

[18] Holdworth, Andrew. Oracle 9i Database Performance Planning, Release 2 
(9.2). Oracle Corporation (Redwood Shores, CA 94065), 2002. 

[19] Kreitman, Kevin B. COTS I GOTS Trade Study Report, Aerospace Teehnical 
Report Number ATR-2005(5154)-3. The Aerospaee Corporation (El Segundo, CA 
90245), 17 December 2004. 

[20] Clinger-Cohen Act of 1996 (divisions D and E of U.S. Public Law 104-106). 

[21] Federal Enterprise Architecture Program Management Office. The Teehnical 
Reference Model Version l.I: A Foundation for Government-wide Improvement, Au- 
gust 2003. 


Prepared Statement of Glenn A. Fine, Inspector General, Department of 

Justice 

introduction 

Mr. Chairman, Senator Leahy, and Members of the Subcommittee on Commerce, 
Justice, State and the Judiciary: 

I appreciate the opportunity to testify before the Subcommittee as it examines the 
Federal Bureau of Investigation’s (FBI) Trilogy information technology (IT) mod- 
ernization project. The Trilogy project was designed to upgrade the FBI’s IT infra- 
structure and replace its antiquated case management system with the Virtual Case 
File (VCF). 
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Successful implementation of the Trilogy project is essential to modernizing the 
FBI’s inadequate information technology systems. The FBI’s systems currently do 
not permit FBI agents, analysts, and managers to readily access and share case-re- 
lated information throughout the FBI. Without this capability, the FBI cannot per- 
form its critical missions as efficiently and effectively as it should. 

In March 2004, this Subcommittee held a hearing on the status of the Trilogy 
project, and I testified about the schedule delays and cost increases of the Trilogy 
project. At that time, I stated that I was skeptical about the FBI’s proposed sched- 
ule to deploy a fully functional, complete version of the VCF before the end of cal- 
endar year 2004. Shortly before the hearing, the Office of the Inspector General 
(OIG) initiated a follow-up audit to assess the FBI’s management of the Trilogy 
project. 

Today the OIG released the results of this follow-up audit. Our audit found that 
the FBI successfully has completed the Trilogy IT infrastructure upgrades — albeit 
with delays and significant cost increases. However, the FBI has failed to complete 
and deploy the VCF, the critical component of Trilogy that was intended to provide 
the FBI with an effective case management system. The VCF still is not operational 
after more than 3 years of development and the allocation of $170 million. We found 
that the VCF either will require substantial additional work or need to be scrapped 
and replaced by a new system. Moreover, the FBI has not yet provided a realistic 
timetable or cost estimate for implementing a workable VCF or a successor system. 

Our audit also examined the causes for the delays and cost increases in the Tril- 
ogy project. Among the problems were poorly defined and slowly evolving design re- 
quirements for Trilogy, weak IT investment management practices at the FBI, 
weaknesses in the way contractors were retained and overseen, the lack of manage- 
ment continuity at the FBI on the Trilogy project, unrealistic scheduling of tasks 
on Trilogy, and inadequate resolution of issues that warned of problems in Trilogy’s 
development. 

In this statement, I describe the OIG’s examination of the Trilogy project. The 
statement is organized into five parts. First, I provide a brief description of prior 
OIG assessments and testimony about the FBI’s IT systems in general and Trilogy 
in particular. Second, I provide background information on the Trilogy project. 
Third, I discuss the results of the OIG’s recently completed audit regarding Trilogy’s 
cost increases and schedule delays. Fourth, I discuss the OIG’s assessment of the 
causes for the problems in Trilogy’s development and implementation. And fifth, as 
requested by the Subcommittee, I conclude my statement by briefly highlighting 
several ongoing and recently completed OIG reviews that examine a variety of other 
issues in the FBI. 

PRIOR OIG REVIEWS OF FBI INFORMATION TECHNOLOGY 

In a series of reviews over the past several years, the OIG has identified problems 
in the FBI’s IT systems, including outdated infrastructures, fragmented manage- 
ment, ineffective systems, and inadequate training. 

For example, a July 1999 OIG review examined the actions of the Campaign Fi- 
nance Task Force that investigated allegations of improper fundraising practices 
during the 1996 Presidential campaign. The Task Force relied on the FBI’s anti- 
quated case management system, the Automated Case Support (ACS) system, and 
other FBI databases to obtain information on the individuals and organizations that 
had become subjects of the investigation. In this review, the OIG noted that defi- 
ciencies in the ACS system and the way search results were handled within the FBI 
resulted in incomplete data being provided to the Task Force. 

Another OIG review issued in March 2002 examined how the FBI had failed to 
turn over to defense attorneys hundreds of FBI documents that should have been 
disclosed prior to the trials of Timothy McVeigh and Terry Nichols. The OIG again 
concluded that the FBI’s computer systems were antiquated, inefficient, and badly 
in need of improvement. We found that the ACS could not handle or retrieve docu- 
ments in a useful, comprehensive, or efficient way, and it did not provide FBI em- 
ployees with the type of support they need and deserve. 

An OIG audit issued in December 2002 examined the FBI’s IT investment man- 
agement practices. This audit concluded that that the FBI had not effectively man- 
aged its IT investments because it had failed to: (1) effectively track and oversee 
the costs and schedules of IT projects; (2) properly establish and effectively use IT 
investment boards to review projects; (3) inventory the existing IT systems and 
projects; (4) identify the business needs for each IT project; and (5) use defined proc- 
esses to select new IT projects. We concluded that the FBI continued to spend hun- 
dreds of millions of dollars on IT projects without adequate assurance that the 
projects would meet their intended goals. Our audit made eight recommendations 
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with respect to Trilogy, including urging the FBI to establish schedule, cost, tech- 
nical, and performance baselines and track significant deviations from these base- 
lines. 

In a September 2003 audit, the OIG examined the FBI’s implementation of the 
OIG’s prior IT-related recommendations. While we found that the FBI had made 
substantial progress by implementing 93 of 148 total recommendations, we con- 
cluded that full implementation of the remaining recommendations was needed to 
ensure that the FBI’s IT program effectively supported the FBI’s mission. 

As noted above, in March 2004 this Subcommittee held a hearing to examine In- 
formation Technology in the FBI, at which the FBI Director testified about the sta- 
tus of the FBI’s Trilogy project. At that hearing, the FBI stated that it planned to 
have “a network with Full Site Capability by late spring” and that it was “closing 
in on the goal of completion” of the Trilogy project. 

The OIG initiated our follow-up audit to assess the FBI’s management of the Tril- 
ogy project. In December 2004, the OIG completed a draft of this audit report and 
concluded that the VCF was not operational after more than 3 years of development 
and the obligation of $170 million, and the FBI did not know when the VCF or a 
replacement system would be implemented. 

Pursuant to our standard practice, in late December 2004 the OIG provided the 
draft audit report to the FBI for its response. In early January 2005, the FBI pub- 
licly acknowledged problems and delays in the development of the VCF. In a written 
response to our audit report dated January 26, 2005, the FBI acknowledged that 
the VCF had not met its goals with respect to development of an automated case 
management system. Nevertheless, the FBI stated that the “VCF project remains 
the highest IT priority for the FBI.” 

After receiving the FBI’s comments, the OIG completed this audit report and re- 
leased it today. 

I will now provide background on the Trilogy project and the VCF before summa- 
rizing the main findings of our audit. 

BACKGROUND ON TRILOGY 

Trilogy is the largest of the FBI’s IT projects. As originally designed, the Trilogy 
project had three main components: (1) the Information Presentation Component 
(IPC) — which was intended to upgrade the FBI’s hardware and software; (2) Trans- 
portation Network Component (TNC) — which was intended to upgrade the FBI’s 
communication networks; and (3) User Applications Component (UAC) — which was 
intended to replace the FBI’s most important investigative applications, including 
the ACS, the FBI’s antiquated case management system. Among its major short- 
comings, the ACS does not permit FBI agents, analysts, and managers to readily 
access and share case-related information throughout the FBI. Without this capa- 
bility, the FBI cannot efficiently bring together all of the investigative information 
in the FBI’s possession to solve crimes or help prevent future terrorist attacks. 

The first two components of Trilogy provide the infrastructure needed to run the 
FBI’s various user applications, while the UAC was intended to upgrade and con- 
solidate the FBI’s investigative applications. After the September 11 attacks, the 
FBI decided to replace the ACS with an entirely new case management system, the 
VCF. 

It is important to note that Trilogy was not intended to replace all 42 of the FBI’s 
investigative applications or the FBI’s approximately 160 other non-investigative ap- 
plications. Rather, Trilogy was intended to lay the foundation so that future en- 
hancements would allow the FBI to achieve a state-of-the-art IT system that inte- 
grates all of the agency’s investigative and non-investigative applications. 

Our audit found that in late April 2004, the FBI completed the first two compo- 
nents of the Trilogy project. The FBI deployed new hardware and software, includ- 
ing 22,251 computer workstations, 3,408 printers, 1,463 scanners, and 475 servers, 
and it installed new communications networks. 

However, as I describe in the next section of this statement, this deployment was 
not done as quickly as the FBI hoped or expected. Despite the fact that after the 
September 11 attacks Congress appropriated the FBI an additional $78 million to 
accelerate deployment of Trilogy’s infrastructure components, the FBI completed the 
two infrastructure components by late April 2004, just before the FBI’s original tar- 
get date of May 2004. Consequently, the FBI missed by some 22 months the comple- 
tion date for the two infrastructure components under the accelerated schedule 
funded by Congress. In addition, the total costs for the infrastructure components 
of Trilogy increased from $238.6 million to $377 million over the course of the 
project. 
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And while the infrastructure components are now in place to support improved 
investigative applications, the FBI still is far from implementing the third compo- 
nent of Trilogy, the VCF. 

RESULTS OF OIG AUDIT OF TRILOGY PROJECT 


Trilogy Costs 

Trilogy originally was planned in 2000 as a 3-year, $380 million project. Over its 
life. Trilogy has become a $581 million project that has suffered a continuing series 
of missed completion estimates and associated cost growth. 

Initially, in November 2000, Congress appropriated $100.7 million for the first 
year of the project. In May 2001, the FBI hired DynCorp (which later merged into 
Computer Sciences Corporation (CSC)) as the contractor for the IPC/TNC infrastruc- 
ture components of Trilogy. At that time, the scheduled completion date for the Tril- 
ogy infrastructure was May 2004. In June 2001, the FBI hired Science Applications 
International Corporation (SAIC) to develop the user applications component of Tril- 
ogy (which became the VCF), with a scheduled completion date of June 2004. 

In early 2002, the FBI informed Congress in its Quarterly Congressional Status 
Report that with an additional $70 million in fiscal year 2002 funding, the FBI could 
accelerate the deployment of Trilogy. Congress supplemented the Trilogy budget 
with $78 million from the Emergency Supplemental Appropriations Act of January 
2002, thereby raising projected costs to $458 million. 

In December 2002, the FBI estimated it needed $137.9 million more to complete 
Trilogy, in addition to the $78 million it had received to accelerate completion of 
the project. Congress approved a $110.9 million reprogramming of funds that took 
into account the estimates to complete the IPC/TNC portions of Trilogy, as well as 
an estimate of the costs to complete the UAC portion. The $110.9 million reprogram- 
ming increased the FBI’s total available funding for the project to $568.7 million. 
In addition, $4.3 million for operations and maintenance and $8 million for com- 
puter specialist contractor support were added in fiscal year 2003, for a total of 
$581.1 million — $201 million more than originally estimated. 

The following table describes the cost of Trilogy under the original plan and under 
the current plan: 

[In millions of dollars] 


Component Area 

Original Plan 

Current Plan 

TNC/IPC 

238.6 

337.0 

UAC 

119.2 

170.0 

Contractor Computer Speoialists 

n/a 

8.0 

Integrator 

n/a 

5.5 

Project Management 

22.0 

32.5 

Management Reserve 

n/a 

28.1 

Total 

379.8 

581.1 


Schedule for Trilogy Infrastructure Components 

Despite the increased money provided for Trilogy, its implementation has been de- 
layed significantly. Part of the problem we found was that a stable schedule for Tril- 
ogy never was firmly established for much of the project’s history. Beginning in 2002 
the FBI’s estimated dates for completing the Trilogy project components began to 
swing back and forth and were revised repeatedly. 

The original completion date for deploying the Trilogy infrastructure (the first two 
components of Trilogy) was May 2004. After the September 1 1 attacks, the FBI rec- 
ognized the urgency of completing the project and moved up the completion date for 
deploying the Trilogy infrastructure to June 2003. Later, the FBI said the infra- 
structure would be completed by December 31, 2002. Still later, the FBI informed 
Congress that with an additional $70 million it could accelerate deployment of Tril- 
ogy and complete the two infrastructure components by July 2002 and also deploy 
the most critical analytical tools in the user applications component. 

Yet, the timetable for completing the infrastructure components slipped from July 
2002 to October 2002 and then to March 2003. On March 28, 2003, CSC completed 
a communications network, the Wide Area Network, for Trilogy. The FBI reported 
that the Wide Area Network, with increased bandwidth and three layers of security, 
had been deployed to 622 sites. In April 2003, the FBI also reported to Congress 
that more than 21,000 new desktop computers and nearly 5,000 printers and scan- 
ners had been deployed. 
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In April 2003, the FBI and CSC agreed to a statement of work for the remaining 
infrastructure components of Trilogy, including servers, upgraded software, e-mail 
capability, and other computer hardware, with a completion date of October 31, 

2003. In August 2003, CSC informed the FBI that the October 2003 completion date 
would slip another two months to December 2003. In October 2003, CSC and the 
FBI agreed that the December 2003 date again would slip. In November 2003, the 
General Services Administration (whose Federal Systems Integration and Manage- 
ment Center, known as FEDSIM, had awarded contracts for Trilogy on behalf of the 
FBI) formally announced that CSC had failed to meet the deadline for completing 
work on infrastructure portions of Trilogy that were required to support the VCF 
user application under development. 

On December 4, 2003, CSC signed a commitment letter agreeing to complete the 
infrastructure components of the Trilogy project by April 30, 2004, for an additional 
$22.9 million, including an award fee of over $4 million. An award fee is used when 
the government wants to motivate a contractor with financial incentives. The FBI 
covered these additional costs by reprogramming funds from other FBI appropria- 
tions. In January 2004, the FBI converted the agreement with CSC to a revised 
statement of work providing for loss of the award fee if the April 30, 2004, deadline 
was not met. In addition, the revised statement of work provided for cost sharing 
at a rate of 50 percent for any work remaining after the April 30 deadline. 

CSC met the revised deadline of April 30, 2004, for completing the two infrastruc- 
ture components of Trilogy. As a result, the FBI met the original target set in 2001 
for the infrastructure components of Trilogy, but missed the accelerated schedule 
funded by additional money from Congress by some 22 months. 

Schedule for the Virtual Case File 

In June 2002, the FBI decided to deploy the VCF user application component of 
Trilogy in two phases under an accelerated plan: delivery one in December 2003 and 
delivery two in June 2004. A third delivery eventually was added, also for June 

2004. Delivery one was supposed to consist of the initial version of the VCF, which 
was intended to be a completely new case management system with data migrated 
from the ACS. The VCF also was intended to serve as the backbone of the FBI’s 
information management systems, replacing paper files with electronic case files. 
Deliveries two and three under the contract were supposed to consist of enhance- 
ments and additional operational capabilities to the VCF. 

SAIC provided the first version of the VCF to the FBI in December 2003, in ac- 
cordance with the accelerated schedule. However, the FBI did not accept that 
version because the FBI said it was not a functional system and did not meet the 
FBI’s requirements. Deliveries two and three never occurred because of the difficul- 
ties experienced in completing the initial version of the VCF. The FBI informed the 
OIG that these deliveries are not being pursued now given the problems in the first 
delivery and the FBI’s plans to seek a common interagency platform for a case man- 
agement system (the Federal Investigative Case Management System or FICMS, 
which is discussed below). 

In fact, the FBI has abandoned the intended three VCF deliveries and instead an- 
nounced a new two-track approach for continuing development of the VCF. Track 
one, which the FBI refers to as the “Initial Operational Capability,” includes a 6- 
week test of an electronic workflow process scheduled to be completed by March 

2005. During this test, the FBI’s New Orleans field office and a smaller resident 
agency office will enter investigative lead and case data into a prototype VCF file 
system, and this information will be approved electronically and uploaded into the 
ACS. The FBI intends to obtain user comments on, and assess the performance of, 
this new workflow system being tested in track one. 

However, it is important to make clear that the version of the VCF being tested 
in track one will not provide the FBI with the case management applications as en- 
visioned throughout the Trilogy project because it represents just one developmental 
step in the creation of a fully functional investigative case management system. It 
does not offer full case management capabilities. Rather, it is designed to dem- 
onstrate that documents can be approved electronically and uploaded into the exist- 
ing, obsolete ACS. 

The second track, called Full Operational Capability, is intended to reevaluate 
and update requirements for the next phase of developing a functional case manage- 
ment system to replace ACS. In track two, the FBI plans to identify user activities 
and processes for creating and approving documents and managing investigative 
leads, evidence, and cases. As a result of the information gleaned during track two, 
the FBI is updating and confirming the case management requirements and evalu- 
ating whether currently available software can be adapted for a case management 
system rather than creating a completely new system. 
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In commenting on the findings in our audit report about the delays in the VCF, 
the FBI stated that “In many ways, the pace of technological innovation has over- 
taken our original vision for VCF, and there are now products to suit our purposes 
that did not exist when Trilogy began.” This suggests that the current VCF effort 
may be obsolete and that the FBI may implement an entirely new system to replace 
it. 

Moreover, our audit found that the FBI still does not have a clear timetable or 
prospect for completing the project. The VCF case management application was in- 
tended to replace the ACS and be the sole system within the FBI that would contain 
all investigative lead and case file information in a paperless system. Due to the 
failure to complete the VCF, the FBI continues to lack a modern case management 
system containing complete and accessible investigative lead and case information. 
While the FBI cites in its response to our report advances in other FBI IT systems, 
such as its newly created Investigative Data Warehouse, the VCF case management 
system would have many features that a Data Warehouse does not. The VCF was 
intended to be the backbone of the FBI’s information systems, replacing the FBI’s 
paper case files with electronic files. Case data in the VCF could be approved elec- 
tronically, and the electronic files would be available throughout the FBI imme- 
diately as entered. Various lead and case information easily could be associated for 
analysis. The Investigative Data Warehouse, while perhaps a useful tool, does not 
manage case workflow, does not provide immediate access to case information, and 
does not substitute for an effective case management system. Consequently, the FBI 
continues to lack critical tools necessary to maximize the performance of both its 
criminal investigative and national security missions. 

Federal Investigative Case Management System 

As a parallel effort to the VCF, the FBI recently has stated that it is pursuing 
an effort to develop the Federal Investigative Case Management System (FICMS). 
FBI officials have variously described this effort to the OIG during the course of our 
audit as a continuation of the VCF, a new investigative case management system 
to replace the failed VCF, or a “framework” for the future development of an inves- 
tigative case management system platform. 

In its January 26, 2005, formal response to the OIG audit report, however, the 
FBI stated that the VCF and the FICMS are “two separate, but related projects that 
will move forward simultaneously. The VCF project remains the highest IT priority 
for the FBI, and we are developing an implementation plan that will result in de- 
ployment of a fully functional investigative case and records management system.” 

The FBI also stated in its response that it is continuing to pursue the VCF 
through development of an implementation plan. The FBI hired the Aerospace Cor- 
poration to evaluate currently available software products to determine if they meet 
the FBI’s requirements for a case management system. The FBI also asked Aero- 
space to evaluate the adequacy of the VCF as delivered by SAIC to determine what 
might be salvaged from that effort. 

Yet, the timetable for the FICMS and the VCF still does not appear to be rapid 
or clear. In conjunction with the OIG’s audit, the FBI told the OIG that it hopes 
to award a contract for FICMS by April 30, 2005. But the FBI has not provided its 
estimated costs, a revised schedule for completing the VCF, or a schedule for devel- 
oping a new case management system to replace the VCF through the FICMS effort. 

CAUSES OF trilogy’s PROBLEMS 

We believe the responsibility for ensuring the success of the Trilogy project is 
shared by several parties: the FBI; the Department of Justice; FEDSIM — the compo- 
nent of GSA that awarded Trilogy contracts on behalf of the FBI; and the two con- 
tractors — CSC for the two infrastructure components, and SAIC for the user appli- 
cations component that became the VCF. These entities, to varying degrees, did not 
appropriately contract for, manage, monitor, or implement the Trilogy project. 

In our view, the main responsibility for the problems with Trilogy rests with the 
FBI. The FBI acted on a legitimate and urgent need to upgrade its IT infrastructure 
and replace the antiquated ACS. However, in the FBI’s desire to move quickly on 
the Trilogy project, it engaged FEDSIM to handle the contracting for this very large 
and complex project without providing or insisting upon: defined requirements, spe- 
cific milestones, critical decision review points, and penalties for poor contractor per- 
formance. 

The resulting cost-plus-award-fee contract yielded control to the contactors for de- 
veloping Trilogy’s technical requirements, while leaving the FBI little leverage to di- 
rect the project. In essence, the contract terms required paying the contractors re- 
gardless of whether they met schedules or were even technically capable of com- 
pleting such a challenging project. 
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In addition, the FBI failed to adequately develop and articulate the design re- 
quirements at the outset of the project, and consequently the requirements repeat- 
edly changed as the project progressed, with too much contractor control and too lit- 
tle input from FBI management. 

In its response to the audit report, the FBI alluded to its lack of control over re- 
quirements as a reason for the current VCF problem by stating that “[T]he VCF 
project suffered in part from runaway scope.” The FBI response also stated that to 
guard against runway scope in the future, “the IT system will be designed, devel- 
oped, and deployed incrementally against specified and planned parameters.” 

In addition to the poor choice of contracting method and sketchy requirements, 
neither the FBI, the Department, nor FEDSIM ensured that adequate schedule, 
cost, technical, and performance baselines were established to allow the project to 
be adequately monitored and to identify and rectify schedule slippages or technical 
problems. Since none of the responsible parties ensured that realistic milestones 
were established to complete various segments of the project, it was difficult to en- 
sure that the contractors successfully met overall schedule, cost, technical, or per- 
formance targets for the project. 

In addition, the Department expected the FBI to assume the role of project inte- 
grator to ensure all three Trilogy components meshed properly and were on track, 
even though the FBI lacked this capability or experience. The FBI’s ability to man- 
age the Trilogy project, even with the help of contractor personnel, was crippled fur- 
ther by a revolving door of Chief Information Officers (CIOs) and Trilogy project 
management personnel at the FBI. 

A variety of audits by the OIG and the Government Accountability Office, as well 
as internal FBI reviews, had identified deficiencies in the FBI’s management of IT 
projects, including Trilogy. However, the FBI’s corrective action was slow. Only re- 
cently has the FBI made substantial progress in its IT investment management 
processes. 

More specifically, in our audit report the OIG detailed the following eight causes 
for the FBI’s problems with the Trilogy project: 

— Poorly defined and slowly evolving design requirements. — One of the most sig- 
nificant problems with managing the schedule, cost, technical, and performance 
aspects of the Trilogy project was the lack of a firm understanding of the design 
requirements by both the FBI and the contractors. Trilogy’s design require- 
ments were ill-defined and still evolving as the project progressed. During the 
initial years of the project, the FBI had no firm design baseline or roadmap for 
Trilogy. According to one FBI Trilogy project manager. Trilogy’s scope grew by 
about 80 percent since the initiation of the project. Such large changes in the 
requirements meant that the specific detailed guidance for the project was not 
established, and as a result a final schedule and cost were not established. In 
addition, after the September 11 attacks, the FBI recognized that the initial 
concept of simply modif 3 dng the old ACS would not serve the FBI well over the 
long run. The FBI then created plans for the VCF. Additionally, a need for 
broadened security requirements due to vulnerabilities identified in the 
Hanssen espionage case affected Trilogy’s development. According to one project 
manager, this recognition of the need to upgrade security caused more problems 
and delays for the full implementation of the infrastructure component. 

— Contracting weaknesses. — The FBI’s current and former CIOs told the OIG that 
a primary reason for the schedule and cost problems associated Trilogy was 
weak statements of work in the contracts. According to FBI IT and contract 
managers, the cost-plus-award-fee type of contract used for Trilogy did not re- 
quire specific completion milestones, did not include critical decision review 
points, and did not provide for penalties if the milestones were not met. 

— IT investment management weaknesses. — ^As described in the OIG’s December 
2002 audit report. The Federal Bureau of Investigation’s Management of Infor- 
mation Technology Investments, at Trilogy’s inception and over much of its life, 
the FBI’s IT Investment Management process was not well-developed. Although 
our recent audit found that while the FBI had started centralizing its project 
management structure, appropriate project management was not consistently 
followed by Trilogy’s IT project managers. In essence, the FBI took risks to ex- 
pedite Trilogy’s implementation, and that approach failed because the manage- 
ment practices to oversee Trilogy simply were not in place. 

— Lack of an Enterprise Architecture. — ^An Enterprise Architecture provides an or- 
ganization with a blueprint to more effectively manage its current and future 
IT infrastructure and applications. The development, maintenance, and imple- 
mentation of Enterprise Architectures are recognized hallmarks of successful 
public and private organizations. While the FBI has agreed to develop a com- 
prehensive Enterprise Architecture, this recommendation has not yet been fully 



65 


implemented. The FBI has contracted for an Enterprise Architecture to he com- 
pleted by September 2005. Without a complete Enterprise Architecture, the FBI 
needed to conduct reverse engineering to identify existing IT capabilities before 
developing the infrastructure and user applications requirements for the Trilogy 
project. 

— Lack of management continuity and oversight. — Turnover in key positions hurt 
the FBI’s ability to manage and oversee the Trilogy project. Since November 
2001, 15 different key IT managers have been involved with the Trilogy project, 
including 5 CIOs or Acting CIOs and 10 individuals serving as project managers 
for various aspects of Trilogy. This lack of continuity among IT managers con- 
tributed to the lack of effective and timely implementation of the Trilogy 
project. According to contractor personnel who are advising the FBI on Trilogy, 
the FBI suffered from a lack of engineering expertise, process weaknesses, and 
decision making by committees instead of knowledgeable individuals. 

— Unrealistic scheduling of tasks. — ^Along with the lack of firm milestones in the 
Trilogy contracts, the scheduled completion dates for individual project compo- 
nents were unrealistic. The unrealistic scheduling of project tasks led to a series 
of raised expectations followed by frustrations when the completion estimates 
were missed. According to an FBI official who monitored the development of the 
Trilogy infrastructure. Computer Sciences Corporation had problems producing 
an appropriate work schedule given the resources provided for the project. Until 
the FBI became more active in examining the scheduling of the project, the FBI 
accepted the project’s schedules as presented by the contractor. This acceptance 
began to shift when the FBI’s scheduler worked with the contractor in early 
2003 to establish a realistic work schedule for completing the infrastructure 
components. 

— Lack of adequate project integration. — Despite the use of two contractors to pro- 
vide the three major Trilogy project components, the FBI did not retain a pro- 
fessional project integrator to manage contractor interfaces and take responsi- 
bility for the overall integrity of the final product until the end of 2003. Accord- 
ing to FBI IT managers, FBI officials performed the project integrator function 
even though they had no experience performing such a role. Although FBI and 
Department officials stated that the Department required the FBI to perform 
project integration duties without contractor support, the expertise to ade- 
quately perform this function did not exist within the FIJI. 

— Inadequate resolution of issues raised in reports on Trilogy. — Within a matter 
of months after initiation of the Trilogy project, the FBI recognized significant 
issues that needed resolution. Internal reports issued by the FBI’s Inspection 
Division, Criminal Justice Information Services Division, and consultants iden- 
tified a lack of a single project manager, undocumented requirements, and a 
baseline that was not frozen. Based on internal reports, the FBI was aware of 
the risks that it faced during the development of the Trilogy project. While FBI 
management eventually hired a project manager to oversee the project — a rec- 
ommendation made in all of the reports — the process of defining requirements 
and baselines for the VCF still continues, more than three years after these in- 
ternal reports were issued. Because the FBI did not act timely to resolve the 
findings of these reports, many problems involving project management weak- 
nesses, poorly-defined requirements, and lack of firm targets unnecessarily con- 
tinued throughout much of the Trilogy project’s history. 

I believe it is important to note that, despite the troubled history of the Trilogy 
project, the FBI recently has made some improvements in its management of infor- 
mation technology. One major improvement in the FBI’s IT management was the 
appointment of a new CIO in May 2004 and the consolidation of the FBI’s pre- 
viously fragmented management of IT resources and responsibilities under the Of- 
fice of the CIO. A significant problem in the FBI’s management of IT investments 
was that all of the FBI divisions with IT investments were not under a single au- 
thority and, as a result, had a variety of processes and procedures for developing 
new systems. Under the reorganization, the CIO is responsible for all of the FBI’s 
IT assets, projects, plans, processes, and budgets. 

In December 2004, the Office of the CIO completed an initial version of an IT 
Strategic Plan, which describes how IT will support the FBI’s Strategic Plan and 
mission goals for the next five years. All IT projects now are required to be con- 
sistent with the FBI’s Strategic Plan. 

The Office of the CIO also has developed an FBI-wide Life Cycle Management Di- 
rective to guide FBI personnel on the technical management and engineering prac- 
tices used to plan, acquire, operate, maintain, and replace IT systems and services. 
The directive provides detailed guidance to FBI Program and Project Managers and. 
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if fully and effectively implemented, will help prevent the delays and problems that 
occurred during the Trilogy project. 

As noted above, the FBI also is in the process of creating an Enterprise Architec- 
ture by September 2006. The Enterprise Architecture will provide a blueprint to aid 
the FBI in coordinating and managing its current and future IT infrastructure and 
systems. The FBI also is working on an IT Portfolio Management Program to list 
and technically document all of its IT systems. The FBI anticipates that rec- 
ommendations stemming from its completed IT portfolio will be included in the de- 
velopment of its fiscal year 2007 IT budget. 

In commenting on the OIG’s Trilogy audit report, the FBI cited a number of other 
improvements it has begun to make, such as an IT metrics program to identify and 
measure IT performance, an initiative to standardize and automate IT procurement 
actions, a Program Management Professional certification training program, a Mas- 
ter IT Policy List to coordinate and control IT policies, standardized technology as- 
sessments, and an Information Assurance Program. Further, the FBI told us that 
VCF track one, or Initial Operating Capability, used the FBI’s new IT management 
approach, including identif 3 dng project objectives, requirements, and constraints be- 
fore proceeding to control gates designed to keep the project on track and to regulate 
the release of funds. Also, the FBI said it developed a cost-sharing arrangement as 
part of the renegotiated UAC contract. These initiatives were beyond the scope of 
our audit, and we could not examine the FBI’s claims on these systems. However, 
they appear to represent progress in the FBI’s IT system. But none of them dimin- 
ish the urgent need for the FBI to fully implement a fully functioning case manage- 
ment system like the VCF to create, organize, share, and analyze case information. 

OIG CONCLUSIONS REGARDING TRILOGY PROJECT 

In sum, the FBI has made progress with its management of IT and its implemen- 
tation of the first two phases of Trilogy. Trilogy’s infrastructure improvements have 
been completed, including the delivery of thousands of modern computer 
workstations and other hardware throughout the FBI. Although the Trilogy infra- 
structure improvements were characterized by delays and increased costs, the infra- 
structure now is in place to support improved user applications, including the VCF 
or its successor case management system, which the FBI recognizes as its top IT 
priority. 

Yet, the VCF effort is incomplete, and the prospects for timely completion remain 
unclear. After more than 3 years, multiple missed deadlines, and a price tag of $170 
million, the FBI still does not have an investigative case management system to re- 
place the antiquated ACS system. Further, we are not confident that the FBI has 
a firm sense of how much longer and how much more it will cost to develop and 
deploy a usable system, whether the FBI continues to pursue the VCF system or 
decides to implement a new case management system. 

Finally, we disagree with the FBI’s assertion in its response to our draft report 
that the delays in deploying the VCF and the lack of an adequate case management 
system do not have national security implications. To the contrary, we believe there 
is a critical need to replace the ACS to enable FBI agents and analysts to effectively 
perform the FBI’s mission. The archaic ACS system — which some agents have avoid- 
ed using — is cumbersome, inefficient, and limited in its capabilities, and does not 
manage, link, research, analyze, and share information as effectively or timely as 
needed. While the FBI has made strides in other IT areas — including installing a 
number of systems to share intelligence information and upload numerous docu- 
ments into a data warehouse — the continued delays in developing the VCF affects 
the FBI’s ability to carry out its critical missions. 

ADDITIONAL OIG REVIEWS IN THE FBI 

To conclude this statement, in response to a request from the Subcommittee, I 
summarize briefly the OIG’s ongoing reviews of other priority issues in the FBI. The 
following are examples of ongoing and recently completed OIG reviews that may be 
of interest to the Subcommittee. 

Ongoing OIG Reviews in the FBI 

Terrorist Screening Center. — The OIG is examining the operations of the Terrorist 
Screening Center to determine how it has managed terrorist-related information to 
ensure that complete, accurate, and current watch lists are developed and main- 
tained. 

Implementation of the Attorney General’s Guidelines. — The OIG is reviewing the 
FBI’s compliance with the revised Attorney General Guidelines that govern the use 
of confidential informants; undercover operations; investigations of general crimes, 
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racketeering enterprises, and terrorism enterprises; and warrantless monitoring of 
verbal communications. 

Intelligence Analysts . — The OIG is reviewing the FBI’s recruitment, selection, 
training, and staffing of intelligence analysts. 

FBI’s Handling of the Brandon Mayfield Matter . — The OIG is reviewing the FBI’s 
conduct in connection with the erroneous identification of a fingerprint found on evi- 
dence from the March 2004 Madrid train bombing as belonging to Brandon 
Mayfield, an attorney in Portland, Oregon. 

Alleged Mistreatment of Detainees at Military Detention Facilities . — The OIG is ex- 
amining any involvement of FBI employees in either observing or participating in 
the alleged abuse of detainees at the military’s Guantanamo Bay and Abu Ghraib 
facilities. In addition, the OIG is reviewing when FBI employees reported the allega- 
tions of abuse and how FBI managers handled the employees’ reports. 

The FBI’s Chinese Counterintelligence Program . — At the request of the FBI Direc- 
tor, the OIG is examining the FBI’s performance in connection with the handling 
of Katrina Leung, an asset in the FBI’s Chinese counterintelligence program. 

The Department’s Counterterrorism Task Forces . — The OIG is evaluating the De- 
partment’s counterterrorism task forces to: (1) determine if they are achieving their 
stated purposes; (2) evaluate gaps, duplication, and overlap in terrorism coverage; 
and (3) identify how the performance of each task force is measured. 

Implementation of the Communications Assistance for Law Enforcement Act 
(CALEA ). — The OlG is conducting a follow-up audit of the implementation of 
CALEA, which allows reimbursement to communications carriers for modifications 
of equipment to allow the capability for lawful electronic surveillance. The FBI has 
expended more than $500 million under CALEA. The OIG’s objectives are to review 
the progress and impediments to the FBI’s implementation of CALEA; review 
CALEA’s costs; and determine how the implementation of CALEA has impacted fed- 
eral, state, and local law enforcement in their ability to conduct electronic surveil- 
lance. 

FBI’s Reprioritization Efforts . — The OIG is reviewing how the FBI’s operational 
changes resulting from its reorganization and change in priorities after the Sep- 
tember 11 attacks have affected other law enforcement agencies. 

Recently Completed OIG Reviews in the FBI 

The following are some examples of recently completed OIG reviews related to 
FBI operations: 

— Follow-up Review of the Status of WENT I lAFIS Integration Wecember 2004 ). — 
This OlG review examined ongoing efforts to integrate the federal government’s 
law enforcement and immigration agencies’ automated fingerprint identification 
databases. Fully integrating the automated fingerprint systems operated by the 
FBI and the DHS, known as lAFIS and IDENT respectively, would allow law 
enforcement and immigration officers to more easily identify known criminals 
and known or suspected terrorists trying to enter the United States, as well as 
identify those already in the United States that they encounter. This latest OIG 
report is the fourth in four years that monitors the progress of efforts to inte- 
grate lAFIS and IDENT. 

This OIG report found that while deployment of new IDENT/IAFIS 
workstations to Border Patrol offices and ports of entry represents a significant 
accomplishment, full integration of IDENT and lAFlS has yet to be realized. 
Federal, state, and local law enforcement authorities still do not have complete 
access to information in the IDENT database. Without such access, the FBI and 
DHS fingerprint systems are not fully interoperable, and it is more difficult for 
federal, state, and local law enforcement agencies to identify illegal aliens they 
encounter. 

This OIG report found that the congressional directive to fully integrate the 
federal government’s various fingerprint identification systems has not been ac- 
complished because of high-level policy disagreements among the Departments 
of Justice, Homeland Security, and State regarding such integration. In addi- 
tion, the Department and the DHS still have not entered into a memorandum 
of understanding (MOU) to guide the integration of lAFIS and IDENT. This 
MOU has not been completed because of fundamental disagreements between 
the Department and the DHS over the attributes of an interoperable fingerprint 
system and the number of fingerprints to be taken from individuals by each 
agency. 

— Effects of the FBI’s Reprioritization (September 2004 ). — The OIG reviewed the 
changes in the FBI’s allocation of its personnel resources since the September 
11 terrorist attacks. The report provided detailed statistical information regard- 
ing changes in the FBI’s allocation of resources since 2000. The OIG determined 
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that the FBI has reallocated resources in accord with its shift in priorities from 
traditional criminal investigative work to counterterrorism and counterintel- 
ligence matters. In addition, the OIG review identified specific field offices most 
affected by changes in FBI priorities within various investigative areas, such as 
shifting agent resources from organized crime or health care fraud cases to ter- 
rorism investigations. The OIG report recommended that the FBI regularly con- 
duct similar detailed analyses of its agent usage and case openings to provide 
a data-based view of the status of FBI operations and to assist managers in 
evaluating the FBI’s progress in meeting its goals. 

— Handling of Information Prior to September 11 Terrorist Attacks (July 2004 ). — 
This classified OIG report, conducted at the request of the FBI Director, exam- 
ined the FBI’s handling of intelligence information prior to the September 11 
terrorist attacks. The review focused on the FBI’s handling of an electronic com- 
munication written by its Phoenix Division in July 2001 regarding extremists 
attending civil aviation schools in Arizona, the Zacarias Moussaoui investiga- 
tion, and information related to September 11 terrorists Nawaf al-Hazmi and 
Khalid al-Mihdhar. 

The OIG made 16 recommendations for improving the FBI’s intelligence han- 
dling and counterterrorism efforts, including recommendations targeted towards 
the FBI’s analytical program. The OIG provided the classified version of this re- 
port to the 9/11 Commission and to Congress. In response to requests from 
members of Congress, the OIG is working with the Department to produce an 
unclassified version of this report that can be publicly released. 

— Foreign Language Translation Program (July 2004). — The OIG audited the 
FBI’s translation of counterterrorism and counterintelligence foreign language 
materials. The audit found that the FBI did not translate all the 
counterterrorism and counterintelligence material it collected. The OIG attrib- 
uted the FBI’s backlog of unreviewed material to difficulties in hiring a suffi- 
cient number of linguists and limitations in the FBI’s translation information 
technology systems. The review also found problems in the FBI’s quality control 
program for language translations. The report made 18 recommendations for 
improving the FBI’s foreign language translation program. 

In response to the OlG report, the FBI stated that it plans to implement a 
national integrated statistical collection and reporting system for its translation 
program in fiscal year 2005 that will allow foreign language program manage- 
ment to accurately determine the amount of unreviewed material that needs to 
be translated. The FBI also plans to increase its digital collection systems’ stor- 
age capacity so that unreviewed audio material for critical cases is not deleted 
by the system. In addition, it plans to implement controls to ensure that the 
forwarding of audio among FBI offices via its secure communications network 
is accomplished reliably and timely. The FBI further reported that it plans to 
assess the linguist hiring process, implement measures to reduce the time it 
takes to bring linguists on board, and strengthen quality control procedures to 
ensure that translations are accurate and that all pertinent material is being 
translated. 

— Edmonds Case (June 2004). — The OIG examined the FBI’s actions in connection 
with allegations raised by former FBI contract linguist Sibel Edmonds. Ed- 
monds alleged that her concerns about aspects of the FBI translation program 
were not appropriately handled by the FBI and that her services as a contract 
linguist were terminated in retaliation for her raising these allegations. The 
OIG review concluded that many of Edmonds’ core allegations relating to the 
co-worker had some basis in fact and were supported by either documentary 
evidence or witnesses other than Edmonds. The OIG concluded that the FBI 
should have investigated Edmonds’ allegations more thoroughly. With respect 
to Edmonds’ claim that she was fired for raising these concerns, the OIG con- 
cluded that while Edmonds does not fall within the protection of the FBI’s whis- 
tleblower regulations, Edmonds’ allegations were at least a contributing factor 
in why the FBI terminated her services. 

— DNA Reviews. — During the past year, the OIG completed three reviews exam- 
ining various aspects of DNA laboratories or DNA grant programs. In the first 
review, completed in May 2004, the OIG examined vulnerabilities in the proto- 
cols and practices in the FBI’s DNA Laboratory. This review was initiated after 
it was discovered that an examiner in DNA Analysis Unit I failed to perform 
negative contamination tests. The OIG’s review found that certain DNA proto- 
cols were vulnerable to undetected, inadvertent, or willful non-compliance by 
DNA staff, and we made 35 recommendations to address these vulnerabilities. 
The FBI agreed to amend its protocols to address these recommendations. 
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In a separate review, the OIG audited several laboratories that participate in 
the FBI’s Combined DNA Index System (CODIS), a national database main- 
tained by the FBI that allows law enforcement agencies to search and exchange 
DNA information. The OIG’s CODIS audits identified concerns with some par- 
ticipants’ compliance with quality assurance standards and uploading of unal- 
lowable and inaccurate DNA profiles to the national level of CODIS. 

In a third review dealing with DNA matters, issued in November 2004, the 
OIG audited the Office of Justice Programs’ DNA backlog reduction grant pro- 
gram. This program provides funding to states for the analysis of DNA samples 
collected in cases where no suspect has been identified. The audit found that 
many of the DNA profiles that had been analyzed by the states using grant 
funding had not been uploaded into the FBI’s CODIS system and that grantees 
were not using the funds on a timely basis to reduce DNA backlogs. 


Prepared Statement of Senator Charles Grassley 

Chairman Gregg, I want to thank you and the Ranking Member for holding this 
important hearing on the FBI’s Trilogy project and for allowing me to submit a 
statement for the record. Over the years I have taken a keen interest in making 
sure that the FBI does the best job that it can. Unfortunately, as Inspector General 
Fine has testified today, the Trilogy project isn’t an example of excellence. 

I want to commend General Fine for his report outlining the many problems with 
the FBI’s management of the Trilogy project and it’s Virtual Case File. The results 
of the OIG audit revealed that, although the FBI has completed two of the three 
components of the Trilogy project, the Virtual Case File (VCF) project has failed to 
produce a functioning records management system. More importantly, it seems as 
if the FBI will now actively pursue the development of the Federal Investigative 
Case Management System (FICMS), but has not provided estimated costs for such 
a project or a revised schedule for completing the VCF. 

The audit has determined that the “main responsibility for the problems with 
Trilogy rests with the FBI.” The fact that changes to the system requirements were 
made after the project had been initiated, that contracts were not monitored and 
that project management decisions were made by committees instead of experts 
with a working knowledge of these systems, are all indicative of a plan that had 
failed before it even got off the ground. The absence of an Enterprise Architecture 
and lack of proper scrutiny over the various contracts leads one to believe that 
funds for this project were requested from Congress before a rational and pragmatic 
review of the potential problems were examined. 

Today’s OlG audit is another verse of the same song. On several occasions in the 
last few years, the IG has had opportunity to examine the FBI’s automated case 
support and its IT systems. They have highlighted the flaws and deficiencies and 
made recommendations. As the IG noted in September 2003, the FBI implemented 
many of the IG’s recommendations, but not all of them. Had the FBI fully embraced 
these recommendations the Trilogy project might have been in a different place 
today. In fact one of the problems we see today was noted in December of 2002. At 
that time, the IG concluded that the FBI was spending hundreds of millions of dol- 
lars on I'T projects without adequate assurance that the projects would meet their 
intended goals. Apparently, not much has changed. 

This is particularly troubling, in light of the dire need for a case management sys- 
tem that works. I agree with the IG’s assessment that “there is a critical need to 
replace the ACS to enable FBI agents and analysts to effectively perform the FBI’s 
mission.” Fighting terrorism is the FBI’s main job and not having an adequate ACS 
hinders their effort. The FBI asserts that the failure of the VCF will not impact on 
national security, but frankly, I’d rather not take the chance. Securing the home- 
land is far too important of a task to not have the best tools available. 

After having spent $580 million on the Trilogy project, including $171 million on 
the Virtual Case File, one would think that the FBI didn’t just have the best tools, 
but they have all of the tools. Unfortunately, the taxpayers $171 million was squan- 
dered on a project that doesn’t meet the FBI’s needs. Additionally, the fact that the 
FBI has been set back three years in planning their critical infrastructure, neces- 
sitates a well thought-out and managed solution. I hope that from this failure the 
FBI can gain some insights and build a learning curve that will help them as they 
look for a another system. 

To that end, I would recommend that the FBI explore the case management pro- 
grams already utilized by other federal government agencies, before attempting to 
spearhead a Federal Investigative Case Management System. It is quite possible 
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that a program currently in use by the federal government could be adapted to suit 
the needs of the FBI case management program. 

Chairman Gregg, I again want to thank you for giving me the opportunity to 
weigh in on this critical topic. General Fine, thank you for your thorough and in- 
sightful report and testimony. I really do want the FBI to be the best they can be 
at protecting America’s citizens, and that’s why this report and hearing are so very 
important. The FBI must learn from its mistakes. To not do so could lead to an even 
greater waste of taxpayers’ dollars and increased risk to national security. 

ADDITIONAL COMMITTEE QUESTIONS 

Senator Gregg. But I am going to have to recess this hearing 
and we are going to have to come hack and reschedule the balance 
at another date. I think the time that the Director has given us has 
been exceptional and it might have been a little longer than people 
had expected, but we appreciate his courtesy. 

[The following questions were not asked at the hearing, but were 
submitted to the Department for response subsequent to the hear- 
ing:] 


Questions Submitted by Senator Patrick J. Leahy 
VIRTUAL case FILE 

Question. There is a field test of the VCF Initial Operating Capability currently 
underway in the New Orleans field office. (A) When will that test be completed? 
Who will assess the results of that test and what criteria will be applied? (B) What 
are the Federal Bureau of Investigations (FBI’s) plans for “VCF-lite?” Does it expect 
to use this software in the future? 

Answer. The deployment of the Virtual Case File (VCF) Initial Operating Capa- 
bility (IOC) as a pilot to the New Orleans Field Office, the Baton Rouge Resident 
Agency, the Criminal Investigative Division’s Drug Unit, and the User Advocate 
Unit provides the opportunity to refine workflow business processes, verify workflow 
requirements, quantify workflow efficiency improvements, develop workflow-related 
system deployment processes, and develop workflow-related training processes. Dur- 
ing the pilot, metrics are being collected to quantify the above goals and assess user 
satisfaction. At the end of the pilot activities, the FBI will better understand the 
opportunities an electronic workflow capability provides for improving the efficiency 
of document-related business processes and the challenges involved in deplo 3 dng a 
web-based workflow application across the workforce. Questions the FBI hopes to 
answer include: 

— Will the automation of workflow reduce investigation time? 

— Will the application track documents throughout their existence? 

— What is the impact of the automated workflow on the FBI workforce? 

— What is the best way to train FBI employees on new technology tools and capa- 
bilities? 

— Is the user interface acceptable to the users and does it enhance their ability 
to do their work efficiently and effectively? 

— What is needed to implement more effective security controls to ensure seam- 
less access to data and information sharing? 

— Will the interface between the Automated Case Support (ACS) system and the 
VCF IOC be adequate for future system integration efforts, including “flash cut- 
over” strategies? 

The pilot was completed in late spring of 2005, metrics are being analyzed and 
reported. The FBI has tasked Mitretek Systems to perform the assessment to deter- 
mine what future use of the application is appropriate. The results of this pilot, 
along with the conclusions drawn by the Office of the Chief Information Officer 
(OCIO), will be shared with The RAND Corporation as well as our oversight part- 
ners, including the Congress, Office of Management and Budget (0MB), and Gov- 
ernment Accountability Office (GAO). 

The IOC was developed in two increments: the initial IOC and the incremental 
“IOC Plus.” The incremental IOC Plus consists of a few additional features that 
were identified during the beta test as significantly enhancing the usability of the 
IOC and capable of development and testing for a minimum additional cost. Based 
on feedback from users during the pilot, the FBI will determine the value and cost 
of deploying IOC Plus field-wide. Such a deployment would provide a stop-gap capa- 
bility while the Full Operating Capability solution is being developed. The FBI has 
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prepared a cost estimate for deploying IOC Plus field-wide and associated operations 
for a 12-month period. These costs will be analyzed along with the perceived benefit 
of such a deployment to the user community, resulting in a recommendation of 
whether to deploy IOC Plus field-wide. 

Aspects of the pilot software will be used in the future. In particular, the Web 
ACS capabilities integrated into the pilot are being further developed and will pro- 
vide users increased access to ACS data. In addition, the software implementing the 
workflow aspect of the pilot is under evaluation for longer term, future use. There 
are no plans for a “VCF-lite.” 

Question. You told this subcommittee on March 23, 2004, that in the wake of the 
9/11 attacks, you evaluated whether to develop VCF or purchase a commercial off- 
the-shelf product. You stated: “I have had a number of persons outside the bureau 
look at the decision to develop our own, persons — I call them the gray-beards — who 
are from a number of private concerns who would look at the choice we made and 
the product we’ve come up with. And I think the reviews are very good for the prod- 
uct we’ve come up with.” Did these “gray-beards” from private concerns produce 
written or otherwise formal assessments of whether the FBI should develop its own 
product or buy off-the-shelf? If so, please provide those. If not, why did the FBI 
chose to rely on an informal assessment rather a formal report like the one recently 
prepared by Aerospace? Couldn’t the FBI have benefited by contracting for a formal 
report on off-the-shelf alternatives much earlier? 

Answer. Two groups of “gray beards” reviewed Trilogy and/or VCF. A panel from 
the National Academy of Science (NAS), led by Jim McGrotty, looked at Trilogy first 
in September 2002, and then again during late 2003 and early 2004. The initial 
NAS effort in September 2002 consisted of two days of briefings, after which the 
panel provided an oral assessment to the Director and others. No formal written re- 
port was issued. The purpose of this review was to ^ve the Director a “pulse-check” 
on how Trilogy was proceeding. The second NAS review resulted in a written report, 
issued in May 2004, and was followed by an addendum in June 2004, that focused 
on changes made by Zalmai Azmi, after his appointment as the FBI’s Chief Informa- 
tion Officer (CIO) (which were not considered in the initial report). The NAS was 
not asked to assess commercial off-the-shelf (COTS) products, although some panel 
members believed, from the discussion of requirements presented, that COTS prod- 
ucts might meet FBI requirements. 

The second group of “gray beards” is the Director’s Science and Technology Advi- 
sory Board, led by Art Money. This Board has been briefed on Trilogy, VCF, and 
the FBI’s information technology (IT) effort in general since it first began meeting 
in October 2003. At the request of former FBI Executive Assistant Director Wilson 
Lowery, members of the Board met in June 2004, specifically to review and com- 
ment on the VCF Corrective Action Plan, including the identification of any incon- 
sistencies or gaps in the remediation plan. After a series of briefings during the day, 
the Board members met with the Director to provide an oral assessment of the plan 
and other suggestions. Since then, VCF, Enterprise Architecture (EA), and IT have 
been regular agenda items on the Science Board’s agenda, and the program man- 
agers have updated the board members. Again, the Board was not asked to assess 
COTS, but they also suggested that COTS could satisfy most of the FBI’s require- 
ments and encouraged the FBI to explore that option. 

Question. On July 16, 2002, Sherry Higgins, your then-Project Management Exec- 
utive, testified before the Senate Judiciary Subcommittee on Administrative Over- 
sight that the FBI was using an industry-standard process — Joint Application De- 
velopment — to “define and prioritize” its requirements. The subcommittee was also 
told this was a new way of doing business: bringing together users, designers, and 
future systems operators to define and prioritize requirements. Why was this proc- 
ess not effective in producing a concrete and final list of VCF requirements that 
SAIC could use to build VCF? 

Answer. The Joint Application Development (JAD) sessions resulted in user-need 
and requirements statements reflecting the capabilities users desired in the final 
system. These statements were, however, not prioritized and, in some important as- 
pects, insufficiently detailed (this was particularly true of the requirements related 
to users’ access to the system’s functions and data and to the requirements defining 
the system’s administrative functions). As a result, the requirements were accurate 
and consistent, but they were not complete in all areas. In addition, they did not 
reflect the constraints imposed by the system’s conceptual design or current infra- 
structure, since the process by which requirements were defined was implemented 
after the Science Application International Corporation (SAIC) had already devel- 
oped the conceptual design and the infrastructure framework. The SAIC attempted 
to use these user-need and requirements statements to define a set of requirements 
that were consistent with a vision of how to build the system, but were unsuccessful 
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because the JAD lacked sufficiently detailed requirements regarding security, 
records management, and the intelligence mission to complete the new system and 
application architecture. 

Question. At the hearing on February 3, you described problems leading to a 
failed VCF effort, including that the FBI “lacked skill sets in our personnel, such 
as qualified software engineering, program management and contract management.” 
You also stated that the FBI responded to these deficiencies by outsourcing the con- 
tract management and technology development. The Federal Systems Integration 
Management (FEDSIM) is acting as the contracting office on behalf of the FBI, and 
Mitretek Systems provides program management, systems engineering and tech- 
nical advisory services. SAIC has been responsible for delivering VCF. 

Has the FBI or an outside entity evaluated the extent to which it should have 
such capabilities within its own staff, and if so, what is the result of that assess- 
ment? 

Answer. In 2003, a distinguished group under the NAS conducted an in-depth 
study of the Trilogy program including, in particular, VCF. The NAS determined 
that, while the FBI had some good IT people, it fell short of the kind of expertise 
needed to manage large IT acquisitions, not only from the program management 
perspective, but also from an engineering perspective. 

The FBI recognized these shortcomings and created the Office of the Chief Tech- 
nology Officer (CTO) in June 2004, to strengthen engineering and computer science 
especially as it relates to the development of new technology. Currently, that office 
includes 10 software/system engineers and is in the process of selecting an addi- 
tional 10, which will be a combination of new employees and transfers from other 
parts of the FBI. At the same time, the CTO is stren^hening software, system, and 
data engineering at the Bureau by hiring contractors to work on establishing “to be” 
technical and data reference models for the enterprise, participating in project and 
critical design reviews, and base-lining the FBI’s capabilities from a systems engi- 
neering perspective. The FBI plans to request additional government software and 
systems engineers in the future to bolster its resource pool for dealing with complex 
and critical information technology projects. 

Additionally, the FBI’s Office of IT Program Management (OIPM) has taken sev- 
eral steps to strengthen program management skills as they relate to IT programs 
and projects. In addition to recruiting several experienced program managers to fill 
key IT management positions within the OIPM, the FBI has implemented a training 
initiative through which employees can be certified as Program Management Profes- 
sionals. Since September 2004, 30 employees have been trained and two additional 
classes, with an additional 50 students, are underway. 

On March 4, 2005, the FBI became a member of the Program Management Insti- 
tute (PMI), and enrolled 20 employees in this professional organization. In addition, 
these individuals joined the Washington, D.C. PMI Chapter and the government 
special interest group. This allows FBI program managers to remain updated on the 
latest information from PMI, attend project meetings, and participate in the govern- 
ment-specific interest group. 

Question. What adjustments did the FBI make to its own internal personnel to 
ensure proper oversight of, and effective communication with, SAIC, FEDSIM, 
Mitretek and other entities performing functions related to VCF? 

Answer. While the FBI did take steps to improve internal oversight of the VCF 
project, in retrospect the steps were not adequate to ensure proper oversight. Steps 
taken included, but were not limited to, the following: (1) a communication plan be- 
tween the FBI User Team and SAIC where FBI User Team members were inte- 
grated into SAIC’s environment for project support; (2) an interim Award Fee feed- 
back plan instituted by the FBI and the Federal Systems Integration Management 
(FEDSIM) Center to provide SAIC more frequent performance analysis; (3) monthly 
In-Progress Review (IPR) briefings provided by SAIC to FEDSIM and the FBI; and 
(4) weekly Program Management Office (PMO) meetings, attended by SAIC, 
FEDSIM, and the FBI. 

Once it became apparent that SAIC was having difficulties, the PMO co-located 
FBI and Mitretek Systems personnel at SAIC. Mitretek was asked by the FBI to 
provide additional resources to help resolve issues. In addition to attending ad hoc 
meetings as issues arose, Mitretek Systems developed a User’s Guide and a series 
of white papers to assist SAIC in understanding the FBI’s needs and requirements 
in areas not covered in detail in the system and software requirements. The white 
papers addressed such topics as: access control concepts, authorized users, delegated 
functions identified in the Systems Requirements Document, lead routing table con- 
cepts, package assumptions rules and roles, silent hits, User Application Component 
client server communications link bandwidths, logging, and data models. 
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In the first quarter of 2004, an independent FBI Special Technologies and Appli- 
cations Section technical team provided an architectural evaluation, identif 3 dng 
risks and deficiencies previously suspected, but not confirmed, to the PMO. This 
new risk information, in addition to SAlC’s past performance on this project, became 
an important component in the FBI’s assessment of SAIC’s ability to respond to the 
challenges of completing VCF delivery 1. 

Question. Prior oversight reports have found that the bureau has had trouble 
managing its information technology contractors and that these problems contrib- 
uted in part to cost, schedule, and performance shortfalls on system modernization 
projects such as TrilogyA^irtual Case File. For example, in December 2002, the In- 
spector General reported that the bureau was not implementing cost, schedule, and 
technical baselines to monitor its contractor’s progress on the Trilogy project. In ad- 
dition, in May 2004, the National Research Council reported that the bureau needed 
more control over its TrilogyA^irtual Case File contracts, including more frequent 
use of contractor progress reviews, performance metrics, and specific milestone de- 
livery dates. What has the FBI done to strengthen its ability to effectively manage 
its contractors and minimize the risk that the bureau will experience cost, schedule, 
and performance shortfalls on future IT projects? 

Answer. In May 2004, Director Mueller announced the appointment of Mr. Zalmai 
Azmi as the FBI CIO. Mr. Azmi is responsible for the FBI’s overall information tech- 
nology efforts, including developing the FBI’s IT strategic plan and operating budg- 
et; developing and maintaining the FBI’s technology assets; and providing technical 
direction for the re-engineering of FBI business processes. Mr. Azmi was given the 
authority for enterprise-wide IT budget control/consolidation. The CIO and Chief Fi- 
nancial Officer (CFO) work closely together on all IT financial and budget matters 
(including the IT budgets in all FBI divisions). The CIO and CFO instituted an ac- 
quisition process that required all IT investments to be reviewed by CIO and CFO 
staff. Almost 1,000 acquisitions were reviewed and approved in the last 2 quarters 
of fiscal year 2004. The risks associated with cost, schedule, and contract perform- 
ance are also reduced by the FBI’s close coordination with the Department of Justice 
(DOJ) CIO; the FBI and DOJ CIOs meet regularly to discuss status and address 
issues related to the FBI’s major IT investments. 

The CIO centralized the IT business of the FBI, mostly in an organizational struc- 
ture under his office with a few specialized applications organizationally separate 
but reporting through him (e.g., the Criminal Justice Information Systems Division 
in West Virginia) under the Life Cycle Management Directive (LCMD). The LCMD, 
which fundamentally changes how IT projects are managed in the Bureau, governs 
how IT projects are managed from “cradle to grave” and is consistent with industry 
and other government agency best practices. The LCMD guides FBI personnel on 
the technical management and engineering practices used to plan, acquire, operate, 
maintain, and replace IT systems and services. All IT projects and programs will 
be required to undergo rigorous project and executive level “control gate” reviews 
for each stage, from inception through disposal. There are seven gates, nine phases, 
and 14 key supporting processes in the LCMD. These reviews are the mechanism 
for management control and direction, decision-making, coordination, and confirma- 
tion of successful performance. 

The FBI’s CIO has established a system of review boards through which the IT 
business of the FBI is conducted. 

— The IT Advisory Board ensures new technologies are incorporated into FBI op- 
erations and business practices, ensures decision makers prioritize operational 
technology needs for future development, minimizes duplicate technology busi- 
ness practices to better optimize resources, and resolves conflicts involving IT 
issues among FBI Headquarters divisions. 

— The EA Board ensures IT systems comply with EA requirements, the sup- 
porting system concept, critical design, and disposal reviews. 

— The IT Policy Review Board provides guidance and direction on IT policy mat- 
ters, resolves issues, and develops new policies. 

— The Technical Review Board ensures IT systems comply with technical require- 
ments, leads critical design, and supports deployment readiness and system test 
reviews. 

— The Change Management Board manages IT infrastructure changes, leading de- 
ployment readiness, system test readiness, operational acceptance, and disposal 
reviews. 

— The Investment Management Project Review Board ensures IT systems acquisi- 
tions are aligned with IT policy, strategic plans, and investment management/ 
portfolio management requirements. It leads system concept and acquisition 
plan reviews. 
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These Boards have defined roles/responsihilities within the structured framework 
of the LCMD and operate pursuant to established charters and procedures. The 
LCMD applies to all solution providers, including contractors. In addition, contract 
management is enhanced at the Departmental level by the work of the Department 
Executive Review Board (DERB), which oversees DOJ’s major IT investments, in- 
cluding those led by the FBI. The DERB operates as part of DOJ’s Information 
Technology Investment Management process to provide oversight at the highest 
level. 

Through a newly instituted IT Investment Management process, the CIO is estab- 
lishing control and management of IT project budgeting, working with the CFO in 
the budget formulation process during the fiscal year 2006 budget cycle (the Finance 
Division has asked the CIO to provide an addendum to the fiscal year 2007 budget 
request targeting IT requirements). In addition, the OCIO has increased the over- 
sight of IT projects and programs through the development of IT standard system/ 
project definitions; identification of the FBI IT portfolio of systems, applications, 
programs, and projects; release of the FBI IT Master Project/Programs list; promul- 
gation to all FBI divisions of the LCMD; and purchase of an Enterprise Portfolio 
Management tool and a Project Portfolio Management tool. 

Oversight of IT projects begins with establishing baselines for each project. In 
June 2004, it was mandated that all new projects produce and maintain resource- 
driven MS Project 2002 schedules. These schedules are subject to periodic (weekly, 
monthly, and/or at LCMD gates, depending on the project) review and analysis. All 
pre-existing projects will be required to produce and maintain project schedules, 
which are subject to review and analysis at each of the remaining LCMD gate re- 
views. DOJ has announced their implementation of a congressional mandate that 
all projects of a certain size are required to provide American National Standards 
Institute Earned Value Management Systems data. The Earned Value Management 
(EVM) methodology is a project (investment) management process that effectively 
integrates the investment scope of work with schedule and cost elements for opti- 
mum investment planning and control. The OCIO is in the process of reporting 
EVM data to DOJ in compliance with this mandate. 

Question. When SAIC submitted invoices as it spent taxpayer’s money, what were 
the FBI’s procedures for evaluating: (a) whether those expenditures were permis- 
sible under its contract; (b) whether they were producing the necessary result; and 
(c) whether the timing of those expenses put SAIC on track for timely delivery? Why 
did those procedures fail to identify at an earlier date that SAIC would not be able 
to deliver the expected results on the due date, and what changes have been put 
in place to manage future contracts? 

Answer. In a large system development project such as VCF, the key is the devel- 
opment of a base-lined, resource-loaded network addressing both schedule and re- 
sources. Tracking progress against this resource-loaded network reveals whether the 
money is being spent according to the plan and the development is on track. While 
SAIC had a resource-loaded network, it was not sufficiently milestone driven to ex- 
pose the difficulty they were having completing the system development. Even with 
this weakness, the FBI was aware of the project status. Over the past year, the FBI 
has met with DOJ officials and with House and Senate Committee Members and/ 
or their staffs to address issues regarding the VCF and Trilogy programs. 

The FBI is acutely aware of these deficiencies and has taken proactive steps to 
ensure that they do not recur. As noted above, all projects will be managed in ac- 
cordance with the Life Cycle Management Directive. Earned Value Management 
(EVM) reporting requirements will be mandated on projects of a specified size and 
dollar threshold. A work breakdown structure and a detailed, integrated, event-driv- 
en schedule will be developed and maintained for each project. Project status report- 
ing will be accomplished at both the project and enterprise levels. Project Manage- 
ment Reviews will be conducted at the project level, and “control gate” reviews will 
be conducted at the enterprise level. Appropriately designated boards will oversee 
projects and, through the recent deployment of Worklenz software, all oversight en- 
tities will have the same view into a project’s progress. The oversight process will 
also be enhanced by the monthly entry of key budget and milestone data for all DOJ 
IT projects, including FBI IT projects, into DOJ’s IT Dashboard, which will allow 
the FBI’s and DOJ’s CIOs to view status, EVM metrics, and major developments 
affecting progress. 

While clearly VCF does not provide the capabilities the FBI sought, the “lessons 
learned” from the VCF project management were beneficial. As noted above, the FBI 
has developed the LCMD to impose structure and process in system development, 
and the VCF IOC was executed using this new approach to IT management. Project 
objectives, requirements, and constraints were clearly identified before proceeding to 
each control gate, and “go/no go” criteria were used at major milestones to control 
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the release of funding and to keep the project focused. In addition, a cost-sharing 
arrangement was established as part of the renegotiated User Applications Compo- 
nent contract, and adherence to defined management processes was mandated. As 
a result, the VCF IOC development and deployment was completed on schedule and 
within budget. 

Question. At the hearing, you indicated that the FBI has deferred to DOJ for con- 
sideration of whether the FBI can recoup any funds from SAIC, and if so, how much 
and on what basis. When was this issue deferred for consideration and when do you 
expect to receive an answer? Will you inform the Committee immediately upon re- 
ceipt of this assessment? 

Answer. The FBI referred this matter to DOJ’s Civil Division on February 2, 
2006. The FBI asked the Civil Division to “begin to analyze the facts to assist us 
in determining the appropriate course of action” concerning the possible recovery of 
funds from SAlC. The FBI is continuing to work with the Civil Division and the 
General Services Administration to resolve this matter and determine what future 
action, if any, will be taken. At this point, it is not known when a final determina- 
tion will be made. The FBI will inform the Committee when such a determination 
is made. 

Question. The FBI’s response to the Inspector General’s draft report indicated that 
the FBI established its baseline Enterprise Architecture (EA) in 2004 and is in the 
process of developing a target EA in September 2005. What is the status and 
progress of the bureau’s efforts to develop and implement an effective and complete 
EA that can be used to effectively guide and constrain its IT investments, and will 
it be complete by September 2005? Does it make sense to continue to pursue VCE, 
or even the Federal Investigative Case Management System (FICMS), before this 
EA is complete? 

Answer. Since the award of a contract for EA support on March 21, 2004, the FBI 
has applied a focused, concentrated, and elevated effort. For example, a revised EA 
Program Plan was completed and signed by the CIO on July 2, 2004. EA develop- 
ment efforts and products are being reviewed approximately every other month by 
the Director’s Science and Technology Advisory Board, an external group of senior 
scientists and technology experts. Both completed and in-progress EA products are 
also reviewed by the EA Board (EAB), which includes Deputy Assistant Directors 
from FBI Headquarters Divisions. The EA principles and the Integrated EA Base 
were completed and approved by the EAB and the CIO on December 9, 2004. The 
Integrated EA Base Line, which was approved by the EBI Director on December 21, 
2004, contains the following information. 

— Business Architecture — 36 stakeholders, 42 functions, 223 sub-functions. 

— Data Architecture — identified 8 data areas and 66 data classes. 

— Applications Architecture — includes the Master IT Systems List with an inven- 
tory of over 500 FBI systems, applications, networks, and databases. 

— Technology Architecture — includes the FBI IT Master Products List with over 
800 COTS and Government off-the-shelf products. 

The CIO has added both in-house personnel and contractors to ensure completion 
of the target, or “To Be,” EA by May 2005. The initial phase, referred to as the “in- 
terim To Be architecture,” addresses the target architecture and the Integrated 
Baseline Architecture, focusing on current projects and interoperability within the 
current technology environment. Any project that is being developed with incre- 
mental capabilities will need to be aware of the architectural impact of projects in- 
progress to address integration issues. Phase I of the target architecture identifies 
the mission requirements being supported by the FBI projects identified for fiscal 
year 2005 and 2006, including VCF, and the EA team is working with the personnel 
responsible for the VCF to ensure that its architectural issues are addressed. The 
interim target architecture includes mapping to the reference models identified in 
the Federal EA, tailored for the FBI environment to provide architectural support 
for projects under development. The intent is to create an optimum architecture en- 
vironment for the implementation of projects that enhance the FBI’s technical envi- 
ronment so the FBI is in an optimum position to support the VCF effort. 

Question. Mr. Azmi testified at the hearing that the FBI now has a list of require- 
ments for VCF and has mapped these requirements “through a federal enterprise 
architecture framework,” that these have been broken down into phases, and that 
another independent contractor is assessing the cost of implementing those phases 
and will have a report by mid-February. What is the relationship between this “fed- 
eral enterprise architecture framework” and the FBI’s efforts to develop its own EA 
by 2005? 

Answer. The Federal EA Framework (FEAF) that was initially established in fis- 
cal year 2000 has been evolving with the development of OMB’s five reference mod- 
els. For example, the original FEAF did not contain any framework support for se- 
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curity. Additionally, 0MB and GAO recognized that focusing on specific organiza- 
tions’ applications retained the dependencies or bottlenecks within these organiza- 
tions. Therefore, 0MB replaced that approach with one that employs the concept of 
service components independent of an application’s implementation. Some of the ref- 
erence models, including the Security reference model, are still under development. 
The 0MB approach is to include in the reference models a master list of all possible 
elements, so that an organization can develop its own reference model by selecting 
the elements appropriate to that organization. The FBI is using the 0MB reference 
models to complete its EA to the extent possible, adding additional features or 
framework elements, such as security services and features, as appropriate. The FBI 
target model also uses the reference models, but depicts the future environment the 
FBI expects to need to meet mission goals. The difference between the baseline EA 
and the target EA represents the gap that must be bridged to achieve the target 
EA. 

Question. Is the list of requirements referenced in the hearing the final list of re- 
quirements against which VCF will be built, or will there be additional changes to 
the requirements list, perhaps when the FBI’s own EA is complete in 2005? 

Answer. The FBI contracted with BAE Systems to review and revalidate users re- 
quirements because the mission of the Fill has evolved, presenting new require- 
ments for information and intelligence sharing among different entities. This review 
is still in progress. To ensure future IT systems do not expand beyond their func- 
tional level, IT systems will be designed, developed, and deployed incrementally 
against specified and planned parameters. 

Question. Which independent contractor is developing a cost estimate, and will 
you provide the cost-estimate report to Congress upon receipt? Does it make sense 
to solicit cost estimates at this time given the potential flux in the VCF require- 
ments? 

Answer. Two Independent Government Cost Estimates (IGCEs) are being devel- 
oped, one by Mitretek Systems and one by Aerospace. Aerospace is a Federally 
Funded Research and Development Corporation and is Congressionally chartered to 
provide this kind of analysis. An IGCE is based on a set of assumptions addressing 
the concept and its development, operations, and maintenance. Given the current 
fluidity in the concept’s development, these estimates will need to be revisited when 
the final concept has been defined. 

Due to the rigor and time associated with developing an IGCE, the FBI decided 
to begin preparing the estimates and factor in time for later updates, rather than 
waiting until everything is known to begin to prepare the estimates. 

The FBI would be pleased to brief the Subcommittee on our progress in this area. 

Question. The OIG Report indicates that the FBI discontinued its pursuit of cer- 
tain enhancements and additional operational capabilities to VCF in part because 
the VCF Delivery 1 did not meet its expectations, and also because the FBI plans 
to pursue the Federal Investigative Case Management System (FICMS). The OIG 
Report also indicated that the FBI is serving as the executive agent of the process 
to award a contract for FICMS by April 2005. The FBI’s response to the OIG’s draft 
audit described FICMS as a “blueprint” and stated that VCF and FICMS “are on 
parallel tracks that will eventually converge.” What will this “blueprint” entail and 
who designed it? 

Answer. The Federal Investigative Case Management Solutions (FICMS) initia- 
tive, part of OMB’s Case Management Line of Business, is a framework that pro- 
vides guidance for participating agencies designing and developing investigative 
case management systems. FICMS ensures, where appropriate, the establishment 
and reusability of common IT solutions and promotes the inter-agency compatibility 
and system interoperability needed to facilitate information sharing across the fed- 
eral investigative and law enforcement landscape. Investigative agencies share core 
business functions but also have unique needs that drive agency-specific system re- 
quirements. The FICMS framework uses a Service Oriented Architecture approach, 
which allows agencies the flexibility to implement a common, core solution and build 
specific functional modules that plug into the common solution to meet unique agen- 
cy needs. Accordingly, investigative agencies will procure commercially available so- 
lutions where appropriate, then implement these solutions to address specific activi- 
ties such as investigative workflow management, records management, and data 
analysis. These agency-specific systems will follow the broad FICMS blueprint so 
that data can flow easily and securely between agencies. The FBI is planning to im- 
plement the first investigative case management system as part of the FICMS 
framework, and is collaborating with DOJ and the Department of Homeland Secu- 
rity (DHS) to maximize the system’s use by other investigative agencies, thus pre- 
venting costly investments in duplicate IT case management systems. 0MB selected 
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DOJ to lead this effort, and the FBI was designated as DOJ’s Executive Agent for 
FICMS development. 

Question. What impact has the development of FICMS had on the FBI’s view of, 
or plans for, VCF? What does it mean that VCF and FICMS “are on parallel tracks” 
and “will eventually converge?” 

Answer. The FBI is continuing to move forward to develop and deploy a case man- 
agement system. At the same time, the lessons learned through VCF will be used 
to help develop the FICMS, a broad blueprint for federal investigative case manage- 
ment systems being led by DOJ. The FBI will use the FICMS framework to develop 
an investigative case management system that will not only meets the Bureau’s spe- 
cific needs, but will also provide a blueprint for other federal investigative agencies 
implementing case management systems. The use of this common FICMS frame- 
work will permit more seamless information sharing. 

Question. Will FICMS benefit from the 3-years and $170 million devoted to the 
VCF effort, and if so, in what ways? 

Answer. Yes, the lessons the FBI has learned in its efforts to develop VCF will 
help in developing the FICMS, particularly in the areas of contract management, 
project management, the development and implementation of policies and proce- 
dures, modular development and deployment, data security, records management, 
and training. For example, the FBI learned that it should not attempt a “flash cut- 
over” (i.e., a full implementation of a system in which all functionality is brought 
online initially) when migrating from the legacy system to the new system. Instead, 
the FBI should develop and incrementally deploy capabilities in phases. Also, busi- 
ness process requirements captured through the JAD sessions will be used in the 
development of the FICMS requirements. The electronic interfaces developed be- 
tween the legacy ACS application and VCF IOC are being evaluated for possible 
reuse. The metrics and lessons learned from the New Orleans Pilot, which are cur- 
rently being compiled, will also influence the development of FICMS. 

Question. How will FICMS relate to the FBI’s enterprise architecture? What steps 
has the FBI taken to ensure that these efforts will interrelate, rather than conflict? 

Answer. The FBI is using the FEAF as the basis for the development of the FBI 
EA. 0MB requires that federal agencies use the FEAF, which will ensure interoper- 
ability between systems and easy information sharing. The FBI will use the Service 
Reference Model of the FEAF as the FICMS framework for delivering services in 
a phased approach to participating federal agencies based on their determined pri- 
ority. Each phase will deliver capabilities independently. 

Question. Is there a defined list of requirements for FICMS, such that soliciting 
contracts for FICMS in April will be an efficient and productive process? 

Answer. The goal of this program is to ensure compatibility between all systems 
used by the various entities in DOJ and DHS. In order to ensure that all technology 
requirements will be included in the system’s overarching framework, the FBI sent 
system requirements to DOJ and DHS for review. DOJ and DHS responded by pro- 
viding additional requirements that are necessary for their operations. Based on 
this input, the FBI created a larger set of requirements encompassing the needs of 
the FBI, DOJ, and DHS. This approach ensures that all components’ investigative 
needs be addressed by the framework. 

Question. Besides DHS, what other departments and agencies will FICMS serve? 

Answer. FICMS will serve as a framework for investigative information tech- 
nology systems used by the FBI, DOJ (including DOJ components), and DHS. 

Question. At the hearing, you stated that the FBI “did not have a complete set 
of defined VCF requirements when the original contract was signed in June 2001, 
and we did not have a finalized set until the summer of 2002.” In addition, FBI 
CIO, Zal Azmi testified that “we have completed our requirements. We have a re- 
quirements document for a case management system that our users, our agents, our 
analysts want and the FBI. We have mapped those requirements to our services 
that are guidelines by the federal enterprise architecture framework.” However, the 
Inspector General’s recent audit stated that “the process of defining requirements 
and baselines for the VCF still continues,” and recommends that the FBI “freeze the 
critical design requirements for the case management system before initiating a new 
contract.” Can you reconcile these statements? Are the requirements for VCF now 
frozen and final until a case management system is delivered? How can the VCF 
requirements be final when the FBI does not have a complete EA? When the re- 
quirements are finalized, will an outside expert evaluate the list of requirements, 
and if so, who and when? 

Answer. The OIG report was written in late 2004. Since that time, the FBI has 
made significant progress in documenting the requirements and Concept of Oper- 
ations (CONOPS) for an enterprise-wide case management capability. In January 
2005, the FBI completed the System Requirements Specifications (SRS) and System 
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CONOPS. The SRS have been revised based on feedback provided through a review 
process, and will be finalized at the end of the review/revision process. The 
CONOPS is also undergoing that review/revision process. A System Requirements 
Review for completeness is also ongoing and, after all inputs are incorporated, the 
final set of system requirements will include approval by each of the Lines of Busi- 
ness owners. The systems engineering team is working with the EA team to ensure 
system requirements meet EA objectives. Additionally, requirements will be put 
under formal Configuration Management control. Requirements will be base-lined 
at contract award and, after contract award, changes or proposed changes to the 
system or requested functionality will be managed in accordance with the Configu- 
ration Management Key Support Process of the Life Cycle Management Directive. 
The requirements have additionally been presented to the Director’s Science and 
Technology Board. 

Question. You testified at the hearing that agents will have “a basic case manage- 
ment system” in their hands within a year. What specifically will a “basic case man- 
agement system” entail and will its delivery complete the VCE project? If for some 
reason developments threaten to delay delivery beyond 2005, will you inform this 
subcommittee immediately? 

Answer. The EBI has expended significant time and effort since the hearing con- 
firming requirements for a new case management system, as well as developing a 
procurement strategy that will take advantage of off-the-shelf products. At this time 
the FBI envisions the deployment of the new case management system in four 
phases, each of which will provide discrete aspects of the new case management sys- 
tem. The first phase should be completed 9 to 12 months after contract award, 
which is expected in the summer of 2005. However, we would not expect a “basic 
case management system” to be in place until the completion of phase 2, which will 
not be until 2006. Phases 3 and 4 will add additional capabilities to the system. 

Question. In your testimony, you stated that within 6 to 8 weeks you would have 
an assessment of: (a) the costs required to get a fully functional case management 
system in the hands of agents; (b) the extent to which those costs would require ad- 
ditional funding or reprogrammed funds; and (c) what other programs would lose 
funds, if reprogramming was required. Please provide these assessments to the sub- 
committee immediately upon completion, or apprise us if they will be delayed be- 
yond 8 weeks. 

Answer. The estimate referred to in this question will be based on the IGCEs dis- 
cussed in response to question 9(C), above. The FBI will keep the subcommittee in- 
formed. 

Question. In your testimony, you described the FBI’s development of an Investiga- 
tive Data Warehouse (IDW). Was any part of the development of IDW funded out 
of the $581 million appropriated for the Trilogy project? 

Answer. Funds appropriated for Trilogy were not used for the development of the 
Investigative Data Warehouse. 

TERRORIST SCREENING CENTER 

Question. In December 2003, the FBI’s Terrorist Screening Center began its task 
of consolidating government terrorist watchlists. In the recent White House budget 
submission to Congress, an additional $75 million is directed to the Terrorist 
Screening Center. What is the status of the watchlist consolidation project, and 
what problems have prevented its completion? 

Answer. As of March 12, 2004, the Terrorist Screening Center (TSC) consolidated 
in the Terrorist Screening Database (TSDB) all identifying data from the 12 
watchlists specified in the April 2003 GAO report entitled “Terrorist Watch Lists 
Should Be Consolidated to Promote Better Integration and Sharing” (GAO-03-322). 
While this consolidated database does include the identifying data from the Auto- 
mated Biometric Identification System (ABIS) and the Integrated Automated Fin- 
gerprint Identification System (lAFIS), it does not include the associated purely bio- 
metric information from ABIS and lAFIS. 

Question. When will the government have a complete, integrated terrorist 
watchlist with online access for law enforcement? 

Answer. As noted above, the TSC has a complete integrated terrorist watchlist 
that is now maintained in the TSDB. In addition, information appropriate to perti- 
nent law enforcement groups is exported daily to various information systems, 
where it is electronically accessible to groups that need it in performance of their 
specific duties. Domestically, general law enforcement officers have access to the 
TSDB through the National Crime Information Center system; Customs and Border 
Patrol and Immigration and Customs Enforcement have access through the Inter- 
agency Border Inspection System; the Transportation Security Administration has 
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access through the No-Fly and Selectee lists; and the Department of State has ac- 
cess through the Consular Lookout and Support System. Among the United States’ 
foreign partners, Australian authorities have access through TACTICS, and Cana- 
dian authorities have access through TUSCAN. 

CONCLUSION OF HEARING 

Senator Gregg. I am going to recess the hearing. 

[Whereupon, at 3:20 p.m., Thursday, February 3, the hearing 
was concluded, and the subcommittee was recessed, to reconvene 
subject to the call of the Chair.] 
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